T3DD13 Deployment Workshop – EXT:migrations – (Part 6/10)

This article is part of a series about my deployment workshop on the T3DD13. Make sure to read the other posts, too.

Idea of “migrations” for more then database schema updates

This basic concept of “migrations” for TYPO3 CMS goes back to a talk on the T3DD in 2009 I held together with Christopher. Back then we implemented a Proof of Concept for a customer project and presented that on the devdays.

In that project we had multiple installations (like the live system, a staging sytem, and several developer systems) and needed to apply the same changes to all those independent systems. Those “changes” were things like templavoila TO/DS and mappings, setting up pages with configured plugins, application related configurations in the database and so on. Basically everything one need to do in the backend.

We started to maintain a wiki page and collected all changes (as a step-by-step guide) which need to be done on the next release. Whenever someone updates the codebase on one of the systems, he needs to check the wiki what needs to be done in the backend. This approach was very time-consuming and error-prone. So we wanted to be able to script all these changes and deliver them with the code base in the project SVN (we didn’t used Git back then) and be able to track which changes are already done on one system and which are missing.

From Ruby or Doctrine you might know the concept of migrations. It’s basically made for database schema updates, like creating a new table, adding columns or updating column definitions. A migration provides update from one stage of the database to a new one. A migration can be “applied” (“up” or “execute”) or “removed” (“down” or “rollback”). Each installation “knows”, which migrations are already applied and which are not. With a simple command you can apply all missing migrations (in the correct order) to make sure your database is up to date.

In TYPO3 we have a solution for database schema updates already: The “database compare tool” compares the current database table definitions with those described in the ext_tables.sql files in the extensions. But we can make use of this basic concept to solve the issues described above.

A migration is basically a PHP class with a certain naming convention and in a defined folder. As it is just a file, it can be versionized and distributed to all instances using the VCS (e.g. Git or SVN). Each migration class has a “up”, a “down” method and a unique identifier (a UUID, a timestamp or something else).

Each instance knows which migrations are available (by scanning the defined folders for migration class files) and which of them are already applied. In a backend module and by using a CLI command the user can list and run the missing migrations. This will call the “up” method for each migration and memorize all successfully applied migrations.

As the migrations as just PHP classes, you can do everything you need within the “up” (and the “down”) methods. For example it can use the service classes provided by EXT:coreapi to create/update records.

I see a lot of use cases for such a tool and I’m sure it will help a lot to solve the issue of not-versionized configuration and application related data, which are in the database of TYPO3, but not in files currently.

EXT:migrations

All described above is just ideas and concept. There is no implementation yet. I’m willing to work on a proof of concept and a first implementation. If you like the idea, if you need something like that or have any suggestions, please get in touch with me!

T3DD13 Deployment Workshop – t3xutils – (Part 5/10)

This article is part of a series about my deployment workshop on the T3DD13. Make sure to read the other posts, too.

TYPO3 Extension Utilities

With the refactoring of the core extension manager in one of the last version, it losts the ability to upload extensions to the TYPO3 Extension Repository (TER). By that time I wanted to release an extension in the TER (i think it was the EXT:caretaker), but wasn’t able to do so, because I had no old TYPO3 instance running, which i could utilize for it. I asked myself why one needs a running TYPO3 instance at all, to upload an extension to the TER.

So I started to dig into the extension manager code, which is in charge for up- and downloading extensions to the TER. The result is a standalone PHP script called “t3xutils”.

t3xutils

The script is a full TER client. It can upload extensions to the TER, as well as downloading extensions in a specific version and fetch and show the extension metadata. But it also create and extract .t3x file (archives containing an TYPO3 extension).

It is available on Github https://github.com/etobi/Typo3ExtensionUtils.

deployment?

You might ask: “how is this related to my TYPO3 CMS deployment?”. A valid question.

If you just want to deploy your TYPO3 instance and all of the used extension coming for your or any other repository, you might not need the t3xutils. But for example if you are maintaining an own extension in the TER, and think of automating processes, this can be very helpful (release to TER on tagging in git). Also the ability to fetch and extract extension from the TER using CLI only (e.g. to automate a setup of an new instance and if you don’t have an TYPO3 instance yet) might be a usecase, too.

others and the future of “t3xutils”

Elmar Hinz created an command-line TER client called “EXT”, which is using the “t3xutils” as library for the uploading and wraps it in a comfortable useable cli tool.

Christian Zenker started to refactor the whole t3xutils to base it in the Symphony Console component, which totally makes sense. He has done an awesome job on this so far. His fork is on github, too, and I’m currently working to review and merge it back to my repository and make it the default.

If you are already using, or going to use, the t3xutils, please consider testing the dev-2.0 branch from my repository on github and provide feedback on it. I will appreciate that a lot!