Quantcast
Channel: A TechBlog by Torben Hansen
Viewing all 60 articles
Browse latest View live

Extending Extbase domain models and controllers using XCLASS

$
0
0
In TYPO3 9.5 LTS it has been deprecated (see notice) to extend Extbase classes using TypoScript config.tx_extbase.objects and plugin.tx_%plugin%.objects. In order to migrate existing extensions, which extends another TYPO3 extension, you should now use XLASSes.

For my TYPO3 Extension sf_event_mgt I also provide a small demo extension, which shows how to extend domain models and controllers of the main extension. The previous version using config.tx_extbase.objects can be found here. I migrated this demo extension to use XCLASSes instead.

The code below shows, how two models and one controller are extended using XLASS


// XCLASS event
$GLOBALS['TYPO3_CONF_VARS']['SYS']['Objects'][\DERHANSEN\SfEventMgt\Domain\Model\Event::class] = [
'className' => \DERHANSEN\SfEventMgtExtendDemo\Domain\Model\Event::class
];

// Register extended domain class
GeneralUtility::makeInstance(\TYPO3\CMS\Extbase\Object\Container\Container::class)
->registerImplementation(
\DERHANSEN\SfEventMgt\Domain\Model\Event::class,
\DERHANSEN\SfEventMgtExtendDemo\Domain\Model\Event::class
);

// XCLASS registration
$GLOBALS['TYPO3_CONF_VARS']['SYS']['Objects'][\DERHANSEN\SfEventMgt\Domain\Model\Registration::class] = [
'className' => \DERHANSEN\SfEventMgtExtendDemo\Domain\Model\Registration::class
];

// Register extended registration class
GeneralUtility::makeInstance(\TYPO3\CMS\Extbase\Object\Container\Container::class)
->registerImplementation(
\DERHANSEN\SfEventMgt\Domain\Model\Registration::class,
\DERHANSEN\SfEventMgtExtendDemo\Domain\Model\Registration::class
);

// XCLASS EventController
$GLOBALS['TYPO3_CONF_VARS']['SYS']['Objects'][\DERHANSEN\SfEventMgt\Controller\EventController::class] = [
'className' => \DERHANSEN\SfEventMgtExtendDemo\Controller\EventController::class
];


For domain models, the important part is the registerImplementation() call, since this instructs Extbase to use the extended domain model when an object is processed by the property mapper.

Note, that there are some limitations using XCLASS, so it is highly recommended to read the official documentation.

                       

Extbase $query->statement() - What can possibly go wrong?

$
0
0
Last week I had to resolve a problem in a 3rd party Extension, where an Extbase Plugin returned unexpected results when used multiple times on the same page. The problem showed up in the frontend, where the plugin listed some products by a given category. When the plugin was present one time on a page, the output was as following (simplified):

Output of plugin 1
Product 1 for Category 1
Product 2 for Category 1
Product 3 for Category 1

When the plugin was present two times on a page, the output was as following (simplified):

Output of plugin 1 with Category 1 as selection criteria
Product 1 for Category 1 (uid 1)
Product 2 for Category 1 (uid 2)
Product 3 for Category 1 (uid 3)

Output of plugin 2 with Category 2 as selection criteria
Product 1 for Category 2 (uid 10)
Product 2 for Category 1 (uid 2) <-- Whoops!
Product 3 for Category 2 (uid 11)

Somehow, the output of plugin 2 contained a result, that did not belong to the result set. As written, the examples above are simplified. The output on the production website showed hundreds of products, and just some of them were wrong.

In order to debug the problem, I had a look at the Extbase Repository for the Products Domain model and found this (again simplified).


class ProductRepository extends \TYPO3\CMS\Extbase\Persistence\Repository
{
/**
* @param $categoryUid
* @return array|\TYPO3\CMS\Extbase\Persistence\QueryResultInterface
*/
public function findByCategory($categoryUid)
{
$query = $this->createQuery();
$query->statement('SELECT * FROM tx_products_domain_model_product_' . $categoryUid);
return $query->execute();
}
}

OK... so there are several individual tables for products by category. They all have the same structure and the only difference is, that they have a different name (post-fixed with the category uid) and hold different data. There is also a SQL injection vulnerability, but that has nothing to do with the main problem.

What goes wrong here?


In order to explain, why plugin 2 returns an object, that obviously belongs to plugin 1, you have to know the internals of an Extbase repository, the Extbase QueryResult object and the DataMapper.

Extbase determines the Domain Model based on the Classname. This is done in the constructor of the repository like shown below:

public function __construct(\TYPO3\CMS\Extbase\Object\ObjectManagerInterface $objectManager)
{
$this->objectManager = $objectManager;
$this->objectType = ClassNamingUtility::translateRepositoryNameToModelName($this->getRepositoryClassName());
}

So when the findByCategory function uses the createQuery() function, the query is initialized to create a query for the object type the Repository determined (in this case Product).

When the query is executed using $query-execute(), it returns an object of the type \TYPO3\CMS\Extbase\Persistence\Generic\QueryResult and here we come closer to the explanation of the problem. The QueryResult object has the following function:

protected function initialize()
{
if (!is_array($this->queryResult)) {
$this->queryResult = $this->dataMapper->map(
$this->query->getType(),
$this->persistenceManager->getObjectDataByQuery($this->query)
);
}
}

This function uses the result from the persistenceManager (raw data from the database with language/workspace overlay) and uses the TYPO3 DataMapper to  create an array with Objects of the given type (Product). The DataMapper does this row by row using the following function mapSingleRow($className, array $row)

And here is the final explanation for the behavior of the 2 plugins on the same page.

protected function mapSingleRow($className, array $row)
{
if ($this->persistenceSession->hasIdentifier($row['uid'], $className)) {
$object = $this->persistenceSession->getObjectByIdentifier($row['uid'], $className);
} else {
$object = $this->createEmptyObject($className);
$this->persistenceSession->registerObject($object, $row['uid']);
$this->thawProperties($object, $row);
$this->emitAfterMappingSingleRow($object);
$object->_memorizeCleanState();
$this->persistenceSession->registerReconstitutedEntity($object);
}
return $object;
}

For performance reasons, the DataMapper caches all objects it creates based on their UID. Since the repository in this TYPO3 extension uses different tables (with own UIDs) for data storage, it may happen, that the DataMapper already processed an object with the given UID (but from a different table) and therefore will return a cached version of an object.

So when the output for plugin 1 was created, the DataMapper did create a cached Product object for UID 2 and when the output for plugin 2 was created, the DataMapper returned the cached version of the Product object with UID 2.

So always keep in mind, that an Extbase repository will return objects of exactly one type and that the datasource must always contain unique uids.



How to disable the nginx TYPO3 cache set by ext:nginx_cache in development context

$
0
0
When you run TYPO3 on nginx webservers, you can use the nginx FastCGI cache to really increase the performance of your website. Basically, the nginx FastCGI cache stores pages rendered by TYPO3 in the webservers memory. So once a page is cached in the nginx FastCGI cache, it will be delivered directly from the webservers memory which is really fast.

When using nginx FastCGI cache, the TYPO3 extension nginx_cache can be used to ensure that the nginx FastCGI cache is purged, when TYPO3 content changes. Also the extension ensures, that pages are removed from the cache, when the TYPO3 page lifetime expires.

However, when you do not need the nginx FastCGI cache while developing locally (e.g. no cache configured or even a different webserver), the clear cache function of TYPO3 results in an error message.


In the TYPO3 log you can find messages like shown below:


Core: Exception handler (WEB): Uncaught TYPO3 Exception: #500: Server error: `PURGE https://www.domain.tld/*` resulted in a `500 Internal Server Error`

The message states, that the PURGE request to the nginx FastCGI cache failed, simply because the PURGE operation is not allowed or configured.

Since the extension nginx_cache has no possibility to disable it functionality, you can remove it locally. But if you have the file PackageStates.php in version control, uninstalling the extension can be error prone, since one by accident may commit the PackageStates.php without ext:nginx_cache installed resulting in the cache to be disabled on the production system.

In order to disable the extensions functionality locally (or in your development environment), you should add the following to the ext_localconf.php file of your sitepackage:

// Disable nginx cache for development context
if (\TYPO3\CMS\Core\Utility\GeneralUtility::getApplicationContext()->isDevelopment()) {
unset($GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['nginx_cache']);
unset($GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS']['t3lib/cache/frontend/class.t3lib_cache_frontend_variablefrontend.php']['set']['nginx_cache']);
unset($GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS']['tslib/class.tslib_fe.php']['pageLoadedFromCache']['nginx_cache']);
}

This will unset the cache and all hooks set by the extension if TYPO3 runs in development context. Note, that adding the snippet to AdditionalConfiguration.php will not work, since the ext_localconf.php of ext:nginx_cache is processed after AdditionalConfiguration.php.

Finally, you have to ensure, that the new code is processed after the ext_localconf.php of ext:nginx_cache is processed. In order to do so, you have must add a dependency to ext:nginx_cache the ext_emconf.php of your sitepackage as shown below:

'constraints' => [
'depends' => [
'nginx_cache' => '2.1.0-9.9.99',
],
'conflicts' => [],
]

I should be noted, that the shown technique can also be used to unset/change a lot of other settings which are made by installed extensions.

How to fix TYPO3 error "There is no column with name 't3ver_oid' on table"

$
0
0
Recently the following error message showed up in a project I was updating to TYPO3 9.5:

There is no column with name 't3ver_oid' on table 'tx_news_domain_model_news'.

When you see this message in a TYPO3 project, you should of course first check, if the field is really available in the mentioned table and second, you should check, if extension dependencies are correct.

When extending an Extbase extension (in this case ext:news), you must ensure, that the extending extension is loaded after the extension you extend. In order to do so, you must add a dependency to the extension you extend in your ext_emconf.php like shown below (example when extending ext:news):


'constraints' => [
'depends' => [
'news' => '7.3.0-7.3.99'
],
],

After adding the dependency, make sure to regenerate PackageStates.php ("Dump Autoload Information" in install tool or "composer dumpautoload"  for composer projects)

TYPO3 Extension "Plain FAQ" released

$
0
0
Today I released the first public version of my latest TYPO3 Extension called "Plain FAQ". The name already says it - the extension is a very simple solution to manage Frequently Asked Questions in TYPO3. Below follow some of the facts about the Extension:
  • Compatible with TYPO3 8.7 and 9.5
  • Based on Extbase and Fluid
  • Covered with unit and functional tests
  • Easy usage for editors
  • Uses TYPO3 system categories to structure FAQs by category
  • Field for media and files
  • Possibility to add related FAQs
  • Configurable template layouts for the views
  • Automatic cache cleanup when a FAQ article has been updated in backend
  • Signal slots to extend the extension with own functionality
  • Symfony Console commands to migrate from ext:irfaq
The extension is available on TER and packagist.

Migration from "Modern FAQ (irfaq)"

If you currently use the TYPO3 Extension "Modern FAQ (irfaq)" you may have noticed, that the extension is not compatible to TYPO3 9.5 (last Extension-Update in March 2018)  and the architecture is quite old (AbstractPlugin, Marker Based Templates and TypoScript configuration for wraps). 

For users where "Modern FAQ (irfaq)" is a blocker for an upcoming TYPO3 9.5 Update, it is possible to migrate to "Plain FAQ" using the Symfony Console Commands included in "Plain FAQ". The migration is as easy as the usage of the extension:

1. Migrate existing Categories to sys_category
2. Migrate existing FAQs to "plain_faq" records
3. Migrate existing Plugins including Plugin settings

The migration may not cover all possible scenarios (e.g. Ratings, Question asked by, irfaq Plugin settings set by TypoScript), but is for sure a good starting point in order to migrate existing records. I guess, for most websites the included migration will suite without further work on migrated data.

You can find details about the migration process in the Extension Manual.

Thanks for sponsoring


I would like to thank Julius-Maximilians-Universität Würzburg for sponsoring the initial development of the TYPO3 extension. Thanks for supporting TYPO3 and open source software!







Apache rewrite rule to replace %20-%20 with a dash (#) in URLs

$
0
0
Some old(?) versions of Microsoft Excel replace a dash (#) in an URL with "%20-%20". The following example shows, how Excel transforms URLs:

Original URL:
https://www.domain.tld/some/path/#my-anchor

URL when clicked in Excel:
https://www.domain.tld/some/path/%20-%20my-anchor

This may lead to unexpected behavior on webserver application level e.g. when routing can not be resolved successfully and the request will results in an 404 error.

The probably best way would be to fix this behavior "somehow" in Excel, but this does not always seem to be possible as described in this stackoverflow question.

In order work around this problem for a certain application on a webserver, I added a simple redirect which replaces the "%20-%20" with a "#" using the following .htaccess rewrite rule:

RewriteRule ^(.*)\ \-\ (.*)$ /$1#$2 [NE,L,R=301]

This is for sure not a general solution for the problem, put works perfectly when you only have to fix incoming links for a given application.

How to limit the TYPO3 category tree to a subset of categories for extension records

$
0
0
In many TYPO3 projects I've been working in, the TYPO3 category system is used to structure content by one or multiple categories. A typical category tree I often see is build up as shown in the example below:

Full TYPO3 category tree

This is a very plain way to create a category tree and the structure in the example is limited to 3 independent main categories (Events, Products, Staff). 

Quite often, the shown example category tree is used system wide in TYPO3 and all main categories are shown for all record types. This can be confusing for editors, since when you for example want to assign categories for e.g. event records, why should one see and be able to select the categories "Products" and "Staff" including all subcategories?

Fortunately TYPO3 can be configured to limit the category tree for tables to a given root category. As an example, I limit the shown categories for event records to only "Event" categories. I assume that the category "Events" has the sys_category UID 1.

PageTS example

TCEFORM.tx_sfeventmgt_domain_model_event.category.config.treeConfig.rootUid = 1

In PageTS such configuration options can be set for any record as long as the following configuration path is met: TCEFORM.[tableName].[fieldName].[propertyName]

The fieldName is usually "categories" or "category", but this can also be different depending on how categories are implemented in 3rd party extensions.

The PageTS setting can also be set in TCA as shown below.

TCA example

$GLOBALS['TCA']['tx_sfeventmgt_domain_model_event']['columns']['category']['config']['treeConfig']['rootUid'] = 1;

As a result, the category tree for event records is now limited to the category "Events" and all subcategories.

TYPO3 category tree limited to a subcategories of one main category

I think this is way more clear for an editor than it was before. In general, this can be configured for every table in the TYPO3 TCA (e.g. pages, files, extensions, ...)

The configuration only allows to define one UID as root for the category tree. If more flexibility is needed to limiting the category tree, then TCEFORM.[tableName].[fieldName].[config][foreign_table_where] may be the place to add own custom conditions.

How to add a replacement for the removed TCA Option "setToDefaultOnCopy" in TYPO3 10.4

$
0
0
The TYPO3 TCA Option "setToDefaultOnCopy" has been removed in TYPO3 10 in order to reduce the amount of checks in DataHandler and the amount of available options in TCA. The documentation says, that "This option was only there for resetting some `sys_action` values to default, which can easily be achieved by a hook if needed. If an extension author uses this setting,
this should be achieved with proper DataHandler hooks."

I use this option in one of my extensions. Basically, I have one "main" record, that has one Inline field with various "subrecords". Those "subrecords" are user generated and should not be copied, when the main record is copied, so I had to find out which DataHandler hooks should be used to get the removed functionality back for the TYPO3 10 compatible version of my extension.

After some hours with several breakpoints in the TYPO3 DataHandler I came to the conclusion, that this may not be as "easy" as described, since there is no Hook, where you can unset certain field values during the copy (or localize) process. And if there was, then another problem would have shown up, since relation fields are processed different (basically the relation is resolved using TCA) on copy or translation commands in DataHandler.

Knowing the last about TCA however makes it possible to hook into the process. At a very early stage in DataHandler, I use processCmdmap_preProcess to set the TCA type for affected fields to "none" as shown below:

public function processCmdmap_preProcess($command, $table, $id, $value, $pObj, $pasteUpdate)
{
if (in_array($command, ['copy', 'localize']) && $table === 'tx_extension_table') {
$GLOBALS['TCA']['tx_extension_table']['columns']['fieldname1']['config']['type'] = 'none';
$GLOBALS['TCA']['tx_extension_table']['columns']['fieldname2']['config']['type'] = 'none';
}
}

With this configuration in TCA, the affected fields are completely ignored by the copy/localize command in DataHandler. It is now just important to change the field types back after the command is finished in processCmdmap_postProcess hook as shown below:

public function processCmdmap_postProcess($command, $table, $id, $value, $pObj, $pasteUpdate, $pasteDatamap)
{
if (in_array($command, ['copy', 'localize']) && $table === 'tx_extension_table') {
$GLOBALS['TCA']['tx_extension_table']['columns']['fieldname1']['config']['type'] = 'text';
$GLOBALS['TCA']['tx_extension_table']['columns']['fieldname2']['config']['type'] = 'inline';
}
}

Hard to say, if this is a good approach to get the functionality back. It feels not really right to change existing TCA at runtime as shown, but at least, I could not find any downsides in the solution and it works fine for me.

Unit-, Functional- and Acceptance-Tests for a TYPO3 Extension with GitHub Actions

$
0
0
Back in 2017 at TYPO3 Camp Munich I held a talk about Unit-, Functional- and Acceptance-Tests for a TYPO3 Extension with GitLab CI. I never really used that setup for my Open Source Extensions, since they all are hosted on GitHub. But since november 2019 GitHub Actions are available, so I finally took some time to migrate my GitLab CI Build Pipeline to GitHub Actions. The results of this migration process is available on GitHub and summarized in this blogpost.

To keep things simple, I created a little demo Extension for TYPO3 to make the setup as easy and understandable as possible.

All in all, the results are very satisfying and the build process is really fast without the requirement to use additional docker images (e.g. MySQL or Selenium Standalone). GitHub has really done a great job by providing preconfigured hosted runners with lots of useful tools 👍




The GitHub Repository with all sources and the GitHub Actions workflow is available at https://github.com/derhansen/gha_demo.

During creation of the setup, I ran into some issues, that took me some time to figure out. All issues are easy to resolve and I summarized them in the "Good to know"-section at the end of this article.

TYPO3 demo extension "gha_demo"


The repository includes a very simple TYPO3 extension that basically does nothing special. It has a simple domain model with just one field and a simple plugin that shows all records of the domain model. The extension has the following tests

  • Unit Tests for the domain model
  • Functional Tests for the repository
  • Acceptance Tests (based on codeception) for the plugin output

Before I created the GitHub Actions workflow, I ensured that all tests execute properly in my local development environment.

GitHub-hosted virtual environments


GitHub hosted runners are preconfigured runners that already contain a lot of available software (e.g. composer, PHP in various versions, Google Chrome, Selenium) that can be used to test an application. No need to puzzle around with building or pulling docker images that contain requirements and no waste of build time to install required packages.

For the gha_demo TYPO3 extension I use the Ubuntu 18.04 LTS runner only without any other docker images.

Workflow YAML file


It is very easy to enable GitHub Actions for a repository. You create a directory called .github/workflows and add a YAML file with your workflow configuration that must follow the Workflow Syntax for GitHub Actions.

The workflow YAML file I created for this article is (hopefully) self explaining:
https://github.com/derhansen/gha_demo/blob/master/.github/workflows/ci.yml

The workflow uses 3 GitHub actions. The first action "actions/checkout" just checks out the GitHub repository.

The second action "actions/cache" ensures, that Composer cache files are shared for builds. You just have to configure a unique key for the cache and I choose to use the hash of the composer.json as a key, so every time dependencies change, the cache is rebuilt. To ensure, that the cache is working you should see "loading from cache" in the output of the composer command.



What helps when you want to debug your workflow is to save build artifacts. For this I use the third action "actions/upload-artifact" which uploads the log of the PHP server and the Codeception output if the build failed.

All other steps in the workflow are based on commands that are executed on the runner (e.g. start MySQL Server, Update Composer, ...).

You may note, that the workflow contains 2 "sleep" commands. Both are required so previous commands have enough time to finish execution (start PHP Server and start Selenium).

Another thing you may note is, that I added many TYPO3 packages to the require-dev section of my composer.json file. This is not a requirement and can be moved to an additional build step (e.g. composer require typo3/cms-filelist ....).

Acceptance Tests with Codeception


In order to execute the Codeception Acceptance Tests, it is required to setup a fully working TYPO3 website including a preconfigured database dump with e.g. pages and records to test. For the Acceptance Tests I included the following files/folders in Tests/Acceptance/_data

  • config/ 
    TYPO3 Sites configuration
  • typo3conf/LocalConfiguration.php
    Preconfigured LocalConfiguration PHP that matches the environment and settings (e.g. DB Credentials) for GitHub Actions
  • typo3.sql
    Full Database dump of my local test TYPO3 website

To separate between Acceptance Test environments (local and GitHub) there are configuration settings for both in Tests/Acceptance/_env

At this point I would like to thank Daniel Siepmann for sharing his GitLab CI configuration about Acceptance Tests. I adapted some parts of his examples to my current setup.


Good to know


#1 - Composer dependencies are not cached during builds


Due to a misconfiguration in the Ubuntu 18.04 runner (that has already been fixed), the .composer directory is owned by root:root with 775 rights. This makes it impossible for the runner user to write into that directory. To fix this, make sure to remove the the directory recursive as shown below in a build step before composer is executed.


- name: Delete .composer directory
run: |
sudo rm -rf ~/.composer


#2 - PHP server with "php -S" is obviously not starting


I used "php -S 0.0.0.0:8888 -t .Build/public/ &> php.log.txt &" to start a PHP server that serves my application for Acceptance Tests. Somehow the acceptance tests step was not able to connect to the given port and always showed "Failed to connect to x.x.x.x port 8888: Connection refused”

To solve this issue, I forced the workflow to stop for 2 seconds (just added "sleep 2;" right after the PHP -S line) so PHP has enough time to server the application.


#3 - MySQL credentials not accepted / MySQL "Connection refused"


Setting up a build step that uses MySQL I ran into problems connecting to the MySQL server that comes with the default Ubuntu 18.04 runner. The solution to this problem was really simple, since you just have to start the MySQL service.


- name: Start MySQL
run: sudo /etc/init.d/mysql start

The default credentials for the MySQL are root:root

Testing email delivery of a TYPO3 extension with Codeception, MailHog and GitHub Actions

$
0
0
Some weeks ago I published my blogpost about how to create a GitHub Actions build pipeline for a TYPO3 Extension that executes Unit-, Functional- and Acceptance tests. The extension tested in that blogpost was only a simple demo extension and for me this was a preparation to finally migrate the CI pipeline for my TYPO3 extension sf_event_mgt to GitHub Actions.

The extension comes with lots of unit and functional tests, which are automatically executed on each commit. One missing piece in the puzzle was the automatic execution of my Acceptance Tests, which are based on Codeception and additionally require MailHog in order to test if emails are sent by the extension and if the email content is as expected.

The concept of testing emails in Acceptance Tests using Codeception is really simple. You have to add the composer package ericmartel/codeception-email-mailhog to your dev dependencies and then you are ready to test emails as shown in the abstract of one of my tests below:

$I->fetchEmails();
$I->haveUnreadEmails();
$I->haveNumberOfUnreadEmails(2);
$I->openNextUnreadEmail();
$I->seeInOpenedEmailSubject('New unconfirmed registration for event "Event (reg, cat1) ' . $this->lang . '"');
$I->seeInOpenedEmailRecipients('admin@sfeventmgt.local');
$I->openNextUnreadEmail();
$I->seeInOpenedEmailSubject('Your registration for event "Event (reg, cat1) ' . $this->lang . '"');
$I->seeInOpenedEmailRecipients('johndoe@sfeventmgt.local');

It is also possible to check the email body for various content like I do in other parts of my testsuite.

GitHub Actions supports docker based service containers and MailHog is also available as docker container, so in order to execute my Acceptance testsuite I added MailHog as service container to my CI setup as shown below:

jobs:
  build:
    runs-on: ubuntu-18.04
    services:
      mailhog:
        image: mailhog/mailhog
        ports:
          - 1025:1025
          - 8025:8025

Having the MailHog container in place, the execution of the Acceptance Tests works like a charm. 

Since the Acceptance Tests also cover tests of a plugin that is only accessible by logged in frontend users, the TYPO3 website for Acceptance Tests includes a special configured page with ext:felogin for this scenario. It turned out, that those tests failed on GitHub actions, since Argon2i was not available on the testing runner for whatever reasons. In order to resolve this problem, I configured the TYPO3 website to use BcryptPasswordHash instead of Argon2i which is ok for me, since strong password hashes are not really required in this scenario.

The GitHub actions YAML file is currently available in the development branch of my extension.

The CI results including a downloadable Codeception HTML report for all acceptance tests is available for each build as shown in this example: https://github.com/derhansen/sf_event_mgt/actions/runs/142799855

How to extend existing FlexForm select options of a TYPO3 plugin using Page TSconfig

$
0
0

Sometimes existing options of a TYPO3 plugin may not fully suite the project requirements. As an example, I refer to my TYPO3 extension "Event Management and Registration" (sf_event_mgt). The extension allows to select the ordering of records by a specific field in the FlexForm plugin options as shown on the screenshot below.


The 3 options shown are configured in the Flexform options for the field "settings.orderField".

In a project it was required to order by a custom field which was not part of the main extension. So I added the custom field named "slot" to the extension using an extending extension for sf_event_mgt.

In order to allow the new field as sorting field, the field "slot" needs to be added to the allowed ordering fields using TypoScript (note, this step is only specific to the extension sf_event_mgt).


plugin.tx_sfeventmgt {
settings {
orderFieldAllowed = uid,title,startdate,enddate,slot
}
}

Finally the FlexForm of the plugin needs to be extended, so the new field appears in the "Sort by" select field. In order to do so, the following Page TSconfig has been added:


TCEFORM.tt_content.pi_flexform.sfeventmgt_pievent.sDEF.settings\.orderField {
addItems.slot = slot
altLabels.slot = Slot
}

You might notice the backslash before the dot in "settings\.orderField". This is required to escape the dot of the fieldname "settings.orderField", since Page TSconfig also uses dots to separate between configuration options/levels.

After adding the Page TSconfig, the plugin now shows the new field.

Pretty cool and not a single line of PHP code required :-) 

Reference: TYPO3 TCEFORM

Unexpected sorting behavior after update from MariaDB 10.1 to 10.3

$
0
0

After updating from Ubuntu 18.04 LTS to 20.4 LTS a previously working a PHP application which contains a data export suddenly did not return the expected result any more. I debugged this scenario by comparing the database query results in the data export and obviously, something in the sorting changed from MariaDB 10.1 to MariaDB 10.3

In order to fully reproduce the problem, I created a really simple use case as shown in the SQL dump below.


CREATE TABLE `test` (
`a` int(11) NOT NULL AUTO_INCREMENT,
`b` varchar(255) NOT NULL,
`c` text NOT NULL,
`d` varchar(255) NOT NULL,
PRIMARY KEY (`a`)
) ENGINE=InnoDB AUTO_INCREMENT=6 DEFAULT CHARSET=utf8;

INSERT INTO `test` VALUES (1,'A','\r\n','CRLF'),(2,'A','',''),(3,'A','',''),(4,'A','\r\n','CRLF'),(5,'A','','');

So we have a really simple table structure with some data. The only special thing is, that 2 values in column "c" contain a carriage return and line feed (CRLF). Since this is not printed when selecting data, I also added column d which contains the value "CRLF" for rows, where column c is a CRLF.

So now I select some data. 

SELECT * FROM test;


+---+---+----+------+
| a | b | c | d |
+---+---+----+------+
| 1 | A | | CRLF |
| 2 | A | | |
| 3 | A | | |
| 4 | A | | CRLF |
| 5 | A | | |
+---+---+----+------+

This result is as I would expect it. Now sorting comes into the game...

Ubuntu 18.04 with MariaDB 10.1.47

SELECT * FROM test ORDER BY b ASC, c ASC, a ASC;


+---+---+----+------+
| a | b | c | d |
+---+---+----+------+
| 2 | A | | |
| 3 | A | | |
| 5 | A | | |
| 1 | A | | CRLF |
| 4 | A | | CRLF |
+---+---+----+------+

OK, so the sorting of column c puts the CRLF values at the end for MariaDB 10.1. Now I try the same on another system.

Ubuntu 20.04 with MariaDB 10.3.25

SELECT * FROM test ORDER BY b ASC, c ASC, a ASC;


+---+---+----+------+
| a | b | c | d |
+---+---+----+------+
| 1 | A | | CRLF |
| 4 | A | | CRLF |
| 2 | A | | |
| 3 | A | | |
| 5 | A | | |
+---+---+----+------+

As you notice, the sorting for column c is now reversed...

I did not find a setting in MariaDB 10.3 to switch back to the sorting as it was in MariaDB 10.1. I could also reproduce the same behavior on MySQL 8.0. So... bug or feature - who knows? I think the described scenario can be considered as an edge case, but if you somehow depend on, that sorting for a column with CRLF values is exactly the same, this can hit you really hard.

I created an issue in the MariaDB bug tracker. I'm curious if this is supposed behavior or not.

Replace functionality of TYPO3 extension mw_keywordlist with a custom sitemap content element

$
0
0

One of the big pain points when it comes to TYPO3 major updates are extensions. If the extension stack contains unmaintained / unsupported extensions, updating can really be hard, since existing functionality needs to be replaced and existing data needs to be migrated.

I recently had this problem on a website, where the TYPO3 extension mw_keywordlist (A-Z Keyword List) was used. The project was running on TYPO3 8.7 and was about to be updated to TYPO3 10.4, but an updated version of the extension was not available, so it was a major blocker in the project.

The extension creates a sitemap generated from keywords in the "pages.keywords" field and renders this sitemap in alphabetical order grouped by keyword. So basically it creates just another type of sitemap, which TYPO3 already has other content elements for. In the project team we decided to replace the extension with a custom sitemap content element, which uses a custom dataProcessor to group pages by the configured keywords.

In the sitepackage of the project the following files were added:

  • TCA override for tt_content, so the new content element (CType) "menu_keywordlist" is registered
  • PageTS Config to add the new content element to the content element wizard
  • TypoScript to configure rendering of the new sitemap content element
  • A custom DataProcessor to group pages by keyword
  • A Fluid template to define how the markup is generated
All required files are shown in this GitLab code snippet

After the new sitemap content element has been configured and tested, all existing instances of the mw_keywordlist content element were replaced with the new custom sitemap element. This was done using the following SQL query:


UPDATE `tt_content`
SET `CType` = 'menu_keywordlist'
WHERE `CType` = 'mw_keywordlist_pi1' OR `CType` = 'menu_mw_keywordlist_pi1';

After the existing content elements were replaced, the extension mw_keywordlist could be removed. The new solution was added to the website when it was still running on TYPO3 8.7, since the code is compatible with TYPO3 8.7, 9.5 and 10.4

Thanks to the University of Würzburg for allowing me to share the code of this solution.

How to migrate switchableControllerActions in a TYPO3 Extbase extension to single plugins

$
0
0

TL;DR - I created this TYPO3 update wizard which migrates plugins and Extbase plugin settings for each former switchable controller actions configuration entry.

Since switchableControllerActions in Extbase plugins have been deprecated in TYPO3 10.4 and will be removed in either TYPO3 11 but most likely 12, I decided to remove switchableControllerActions in my TYPO3 Extbase extensions already with the upcoming versions that will be compatible with TYPO3 11.

In this blogpost I will show, how extension authors can add a smooth migration path to their existing extensions by adding an update wizard which migrates all existing plugin settings, so users do not have to change plugin settings manually. 

As a starting point lets have a look at my TYPO3 extension Plain FAQ, which is a very simple Extbase extension with one plugin, that has 3 switchableControllerActions.

  • Faq->list;Faq->detail
  • Faq->list
  • Faq->detail
For all 3 switchableControllerActions, I created 3 individual plugins (Pilistdetail, Pilist, Pidetail) which handle the action(s) of each switchable controller action from the list above. 

For each new plugin, I added an individual FlexForm file which holds the available settings for the plugin. This can be done by duplicating the old FlexForm (Pi1 in this case) and removing those settings, which are not available in the new plugin. Also display conditions based switchableControllerActions must be removed.

Finally I created a new item group for the Plugins of the extension, so all plugins are grouped as shown on the screenshot below.


This is basically all work that needs to be done on code side in order split the old plugin to the new plugins.

Migration of existing plugins


To be able to migrate all existing plugins and settings to the new plugins, I created a custom upgrade wizard that takes care of all required tasks. Those tasks are as following:
  • Determine, which tt_content record need to be updated
  • Analyse existing Plugin (field: list_type) and switchableControllerActions in FlexForm (field: pi_flexform)
  • Remove non-existing settings and switchableControllerAction from FlexForm by comparing settings with new FlexForm structure of target plugin
  • Update tt_content record with new Plugin and FlexForm
As a result, a SwitchableControllerActionsPluginUpdater has been added to the extension. It takes care of all mentioned tasks and has a configuration array which contains required settings (source plugin, target plugin and switchableControllerActions) for the migration.

private const MIGRATION_SETTINGS = [
[
'sourceListType' => 'plainfaq_pi1',
'switchableControllerActions' => 'Faq->list;Faq->detail',
'targetListType' => 'plainfaq_pilistdetail'
],
[
'sourceListType' => 'plainfaq_pi1',
'switchableControllerActions' => 'Faq->list',
'targetListType' => 'plainfaq_pilist'
],
[
'sourceListType' => 'plainfaq_pi1',
'switchableControllerActions' => 'Faq->detail',
'targetListType' => 'plainfaq_pidetail'
],
];
So basically, one configuration entry has to be added for each switchable controller action setting of the old plugin. The wizard determines the new FlexForm settings using configured TCA, removes all non-existing settings (which is important, since TYPO3 will pass every setting available in pi_flexform to Extbase controllers and Fluid templates) and changes the "old" Plugin to the new one.

The update wizard can possibly also be used in other Extbase extensions, since the MIGRATION_SETTINGS are the only configuration options that need to be changed.

The required changes for the complete removal of switchableControllerActions is available in this commit.

How to use constructor dependency injection in a XCLASSed TYPO3 class

$
0
0

Some time ago in needed to extend an Extbase controller in TYPO3 10.4 which used dependency injection through constructor injection. So I used XCLASS to extend the original controller and added an own constructor which added an additional dependency, but this obviously did not work out properly, since the constructor was always called with the amount of arguments from the original class.

Later I created this issue on TYPO3 forge in order to find out if this is a bug/missing feature or if I missed something in my code. In order to demonstrate the problem, I created this small demo extension which basically just extended a TYPO3 core class using XCLASS and just days later, a solution for the issue was provided.

The solution is pretty simple and you just have to ensure to add a reference to the extended class in the Services.yaml file of the extending extension.

Example:


TYPO3\CMS\Belog\Controller\BackendLogController: '@Derhansen\XclassDi\Controller\ExtendedBackendLogController'

The complete Services.yaml file can be found here.

Thanks a lot to Lukas Niestroj, who pointed out the solution to the problem.



"Unterminated nested statement!" using TYPO3 rector

$
0
0

TYPO3 rector is a really helpful application when it comes to TYPO3 major updates. It helps you to identify and refactor TYPO3 deprecations in custom extensions and can save hours of manual refactoring. I use TYPO3 rector quite a lot and stumbled across the following error recently.

"Unterminated nested statement!"







This message is not really helpful, so I digged deeper into the problem. The "Parser.php" throwing the exception is located in "helmich/typo3-typoscript-parser" package, so I first thought that there was a problem with the TypoScript in the desired extension, but after checking ever line manually, I could not find any error. 

It came out, that the extension I wanted to process by TYPO3 rector had a "node_modules" folder, which contained a lot of Typescript (not TypoScript) files. Those files where obviously parsed by the TypoScript parser resulting in the shown error message. After removing (excluding should also work) the "node_modules" folder, everything worked as expected.

If you like rector and/or TYPO3 rector, please consider to support the authors.
 

TYPO3 extension "Event management and registration" version 6.0 for TYPO3 11.5 LTS released

$
0
0

I am really proud and happy to announce, that the new version 6.0. of my TYPO3 extension "Event management and registration"(GitHub / TYPO3 Extension Repository) is now fully compatible with TYPO3 11.5 LTS including support for PHP 7.4 and 8.0.

Originally I wanted to release this version of the extension on the same day as TYPO3 11.5 LTS got released, but I decided to consider all possible deprecations from TYPO3 core and also to reactor the extension to support strict types and strict properties where ever possible. All in all, my planned 6 days for a TYPO3 11.5 LTS compatible version resulted in more than 10 days of work. Well, not all changes were required for the release (e.g. removal of switchableControllerActions), but the code base is now better than before and I'm happy with all improvements that made its way into the extension.

Changes in more than 145 commits

The most important changes are of course those who break existing functionality. Although the new version contains 7 breaking changes and much of the codebase has been changed too, existing users can migrate to the new version with the least possible manual work. 

The list below contains some of the important changes:

  • The extension uses strict types and typed properties wherever possible
  • switchableControllerActions have been removed. The extension now has 7 individual plugins instead. An update wizard will migrate existing plugins and settings.
  • Data Transfer Objects do not extend AbstractEntity any more
  • Native TYPO3 pagination API support for event list view
  • Captcha integration has been refactored to support both reCaptcha or hCaptcha
  • All possible TYPO3 core deprecations have been handled
All breaking changes have been documented in detail in the release notes, so existing users know which parts of the extension need further attention when updating.



How to manually create the default crop variant string for an imported image in TYPO3 CMS

$
0
0

When data in TYPO3 is created automatically (e.g. through a custom API or by an import script), it is very common, that also new files (especially images) are imported. TYPO3 has the well documented FAL (File Abstraction Layer), which provides an API for common tasks. 

One typical task is to import an image to FAL and next creating a file reference for the imported image to a record (e.g. event record of ext:sf_event_mgt). Such a task is easy to implement in a custom extension when you follow the example documentation from the TYPO3 FAL API. This all works fine as described, as long as the referenced image does not use Crop Variants. For image fields, where crop variants are configured, no crop variants will be created for imported images and as a result, TYPO3 will always use the aspect ratio of the imported image no matter which crop variant is configured for output.

TYPO3 internals

Available crop variants for image fields are defined in TCA. When an editor adds an image to a record in the TYPO3 backend, TYPO3 will automatically calculate the default crop variants. The result is saved to the table sys_file_reference in the field crop as a JSON encoded string like the shown example crop variant string below:

{"heroimage":{"cropArea":{"x":0,"y":0.23,"width":1,"height":0.54},"selectedRatio":"16:9","focusArea":null},"teaserimage":{"cropArea":{"x":0,"y":0.14,"width":1,"height":0.72},"selectedRatio":"4:3","focusArea":null}}

This process if performed in ImageManipulationElement. When an editor for example edits a record with an imported image and opens an image inline element, then the ImageManipulationElement is rendered and the default crop variant string is calculated for the imported image and saved, as soon as the editor saves the record.

Manually calculating the default crop variant string

In order to manually calculate the default crop variant string in an image import process, it is required to extract 2 code snippets from the ImageManipulationElement, since both are not public available:

  1. The default crop configuration - see code
  2. The function populateConfiguration - see code
Both is extracted to a class named CropVariantUtility and an additional public function is added which returns the crop variant string for a given file as shown below:

/**
* Returns a crop variant string to be used in sys_file_reference field "crop" for the given file and table/fieldname
*
* @param File $file
* @param string $tableName
* @param string $fieldname
* @return string
* @throws InvalidConfigurationException
*/
public static function getCropVariantString(File $file, string $tableName, string $fieldname): string
{
$config = $GLOBALS['TCA'][$tableName]['columns'][$fieldname]['config']['overrideChildTca']['columns']['crop']['config'];
$cropVariants = self::populateConfiguration($config);
$cropVariantCollection = CropVariantCollection::create('', $cropVariants['cropVariants']);
if (!empty($file->getProperty('width'))) {
$cropVariantCollection = $cropVariantCollection->applyRatioRestrictionToSelectedCropArea($file);
}

return (string)$cropVariantCollection;
}

The whole class is available in this gist.

Finally the new CropVariantUtility can be used in a file import routine as shown in the example below:


protected function addFileToEvent(File $file, int $eventUid): void
{
$eventRecord = BackendUtility::getRecord(self::EVENT_TABLE, $eventUid);

$fileReferenceUid = StringUtility::getUniqueId('NEW');

$dataMap = [];
$dataMap['sys_file_reference'][$fileReferenceUid] = [
'table_local' => 'sys_file',
'tablenames' => self::EVENT_TABLE,
'uid_foreign' => $eventUid,
'uid_local' => $file->getUid(),
'fieldname' => 'image',
'pid' => $eventRecord['pid'],
'show_in_views' => 0,
'crop' => CropVariantUtility::getCropVariantString($file, self::EVENT_TABLE, 'image'),
];

$dataMap[self::EVENT_TABLE][$eventUid] = [
'image' => $fileReferenceUid
];

$this->dataHandler->start($dataMap, []);
$this->dataHandler->process_datamap();
}

The example code will create a new file reference for the given file object for the table  self::EVENT_TABLE (tx_sfeventmgt_domain_model_event in this case) and the given event UID. The usage of the new CropVariantUtility ensures, that the new file relation has a default crop variant string and configured crop variants can directly be used for imported images.

TYPO3 - Multiple dynamic parameters for a typolink using a custom userFunc

$
0
0

I often use the TYPO3 linkHandler to enable the possibility for editors to create direct links to records from within the CKEditor in TYPO3 backend. This is all well documented and easy to configure using the RecordLinkHandler, as long as the resulting link only contains one dynamic parameter. But sometimes it may be required to have multiple dynamic parameters for the resulting link. In this case you may need to create a userFunc for the typolink function in order to create a custom configuration which uses multiple dynamic parameters.

Requirement

Let us assume, you have an event which has multiple event registrations. Registrations are listed in the detail view of an event and each registration is shown as an accordion item with a unique ID in markup. Now you want to create a link to an event and set a link anchor to a specific registration. The resulting URL should be as shown below:

https://www.cool-events.tld/events/my-first-event#registration-1

Calling the URL will open the event detail page and scroll down the the HTML element with the ID "registration-1".

Note: This is just an example, which also can be achieved without a custom userFunc. Goal of this article is to demonstrate how to use a userFunc for typolink.

Solution

In order to archive the requirement, first a linkHandler Page TSConfig must be created as shown below:


TCEMAIN.linkHandler {
event {
handler = TYPO3\CMS\Recordlist\LinkHandler\RecordLinkHandler
label = Event Registration
configuration {
table = tx_sfeventmgt_domain_model_registration
}
}
}

Next, the TypoScript for the link generation is added. 


config {
recordLinks {
registration {
typolink {
parameter = 1
userFunc = DERHANSEN\SfEventMgt\UserFunc\TypoLink->createEventLink
userFunc {
eventUid = TEXT
eventUid.data = field:event
registrationUid = TEXT
registrationUid.data = field:uid
}
}
}
}
}

Finally a custom userFunc needs to be created which renders the A-tag for the link.


<?php

declare(strict_types=1);

namespace DERHANSEN\SfEventMgt\UserFunc;

use TYPO3\CMS\Frontend\ContentObject\ContentObjectRenderer;

class TypoLink
{
private const EVENT_DETAILPID = 22;

public ContentObjectRenderer $cObj;

public function createEventLink(array $content, array $config): string
{
$eventUid = $this->cObj->cObjGetSingle($config['eventUid'], $config['eventUid.']);
$registrationUid = $this->cObj->cObjGetSingle($config['registrationUid'], $config['registrationUid.']);

// Link parameters (can also contain multiple dynamic parameters)
$parameters = [
'tx_sfeventmgt_pieventdetail' => [
'controller' => 'Event',
'action' => 'detail',
'event' => $eventUid,
]
];

$link = $this->cObj->typoLink($this->cObj->lastTypoLinkResult->getLinkText(), [
'parameter' => self::EVENT_DETAILPID,
'additionalParams' => '&' . http_build_query($parameters),
'section' => 'registration-' . $registrationUid,
'returnLast' => 'url',
]);

return '<a href="' . $link . '">';
}
}

The most important part is, that the custom userFunc must only return the opening A-tag. In the userFunc, it is basically possible to construct the resulting link however you want. In the example above, 2 dynamic parameters are used in the function ($eventUid and $registrationUid). It is of course also possible to e.g. do dynamic database lookups in the function to fetch other dynamic parameters required for link construction.


How to use multiple SMTP accounts in one TYPO3 installation

$
0
0

When TYPO3 is used to serve multiple websites in one installation, it may sometimes be required to configure multiple SMTP accounts in order to send emails from TYPO3 (e.g. mailforms or notifications) to different recipients. This may especially be important, when the recipient mailserver has a strict spam filter or when the domain uses a SPF, DKIM or DMARC and the mailserver only accepts emails from thrusted sources.

In TYPO3 you can configure one global SMTP server in LocalConfiguration.php by using the following settings:

$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport'] = 'smtp';
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport_smtp_server'] = 'your.mailserver.tld';
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport_smtp_encrypt'] = true;
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport_smtp_username'] = 'username';
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport_smtp_password'] = 'password';
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['defaultMailFromAddress'] = 'email@your.mailserver.tld';

This setting however reflects to any hosted website in your TYPO3 installation and the email-server for typo3-website1.tld may possible not accept emails with a sender from the domain typo3-website2.tld.

In order to provide multiple SMTP servers for different websites in a TYPO3 installation, I configure different SMTP servers in AdditionalConfiguration.php 


if (($_SERVER['SERVER_NAME'] ?? '') === 'typo3-website1.tld') {
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport_smtp_server'] = 'mail.typo3-website1.tld';
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport_smtp_encrypt'] = true;
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport_smtp_username'] = 'username';
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport_smtp_password'] = 'password';
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['defaultMailFromAddress'] = 'email@typo3-website1.tld';
}

if (($_SERVER['SERVER_NAME'] ?? '') === 'typo3-website2.tld') {
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport_smtp_server'] = 'mail.typo3-website2.tld';
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport_smtp_encrypt'] = true;
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport_smtp_username'] = 'username';
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['transport_smtp_password'] = 'password';
$GLOBALS['TYPO3_CONF_VARS']['MAIL']['defaultMailFromAddress'] = 'email@typo3-website2.tld';
}

Since AdditionalConfiguration.php is evaluated on every request, TYPO3 will conditionally use the email settings depending on the $_SERVER['SERVER_NAME'] variable.

Note, that this solution only applies to web requests and does not work in CLI context (e.g. scheduler task).

Viewing all 60 articles
Browse latest View live