Schlagwort-Archiv: deploy

T3DD13 Deployment Workshop – TYPO3 Surf – (Part 3/10)

This article is part of a series about my deployment workshop on the T3DD13. Make sure to read the other posts, too.

TYPO3.Surf

Surf is an open source tool for deployment, just like Capistrano, Phing or Ant. It’s based on the application framework TYPO3 Flow and thus written in PHP. Even the deployment configurations are written in PHP, so no hassle with ruby scripts or large XML files. Surf can easily be extended by your own Flow package, to add new tasks or even custom deployment workflows. For details about how Surf and the deployment configurations are working, check my article here.

Because Surf is a package for TYPO3 Flow, it can be used „embedded“ to deploy your Flow-based application. You just need to add the TYPO3.Surf package to our application distribution to enable it to deploy „itself“. But you can also setup a „standalone“ Surf instance, which can take care of all your deployments, even if the are not Flow-based. Actually you can deploy anything, which is versionized (currently only Git is supported out of the box, but it easy to add an SVN Checkout task). So it’s no problem to deploy e.g. a TYPO3 CMS or a WordPress, too.

Even if TYPO3.Surf is already used in several production setups, it’s still beta. Some days ago, I met Christopher Hlubek in Kiel at networkteam. We discussed what is missing and needs to be done for a first TYPO3.Surf 1.0 release. There are some easy tasks (like adding a SVN checkout task, some other new tasks or some API cleanups) as well as some meta tasks (seting a product website; provide sample configurations/recipes). I will create a bunch of new issues and cleanup the bugtracker within the next days on forge. If you like to contribute and work on one of these tasks, please get in touch with us!

Custom Surf tasks

During the workshop at the T3DD, we discussed, which custom tasks the participants, which already use Surf, created for theri own needs.

Martin, from n@work, told us about a „reload Apache“-task, which was needed, because the customer virtualhost configuration was versionized in the git repository, too. Beside of that, they created tasks for EXT:coreapi and EXT:t3deploy to clear the caches of a TYPO3 CMS instance or run the database compare tool automatically.

Felix, from Lightwerk, shared some tasks e.g. to create or apply MySQL-dumps, which is pretty helpful, if you want to create for example a testing/dev environment based on the current production database. These tasks, among several other changes, are currently pending for review in Gerrit.

Flexible transfer method

One bigger change in TYPO3 Surf, which was merged recently, are a set of changes bunched toghether under the topic „flexible transfer method“. Before TYPO3 Surf only supported Git to get the code to deploy and additionally the Git repository had to be available from the target node. Surf to connected via ssh to the target node and remotely executed the Git cli command to clone and checkout the repository there. This obviously can’t work, if you don’t have Git installed on the target node. Or, the Git repository might only be available from an internal network and is not reachable from the target node system. To cover these cases, Surf now provides two new stages in the SimpleWorkflow: package and transfer.

In the package stage, Surf can be configured to prepare the code/files locally instead of doing that on the remote host. Surf can do a local git clone, composer update or whatever is needed to prepare the code. Then, during the new transfer stage, the packaged code will be transferred to the remote nodes. This can be done by a rsync, but also with scp or even ftp (in the moment there is only a RsyncTask. But its pretty easy to implement the others, too). Using this new methods the nodes don’t need direct access to the repository anymore. As a side effect, when deploying in a multi-node-environment, your release/code only needs to be prepared/packaged once, instead of on each single node independently. This might speed up your deployment a little.

next steps

The next steps for TYPO3 Surf is to get it ready for a stable 1.0 release as soon as possible. To archive that we could use some help for implementing tasks, refactor some API methods and reviewing/testing changes. Also feedback of people/agencies using TYPO3 Surf to deploy whatever would be highly appreciated.

T3DD13 Deployment Workshop – Deployment Setup – (Part 2/10)

This article is part of a series about my deployment workshop on the T3DD13. Make sure to read the other posts, too.

Example for an deployment setup

During the workshop, we first talked about the deployment setups and tools we use. I presented a setup I recently worked with.

Git Flow

We had all code versionized in Git, using the „Git flow“ approach for branching and releasing. If you don’t know this yet, I highly recommend to read „A successful Git branching model“ and checkout this Git flow cheat sheet. In the end it means we had a stable and always-ready-to-release master branch, a release branch and a develop branch with the latest changes.

Systems

Beside of multiple development installations on the developer’s local systems, we had a live system, a latest and a staging system.

live

On live the latest master branch was checked out (which means the latest stable release). live is also the system where editors editing the actual content.

staging

On staging HEAD of the release branch was checked out. This branch contains the latest changes/features, which are supposed to be released next on live. Once in a while a script (triggered manually, see below) copies the (MySql-) database and asset files (like fileadmin/ and uploads/ using rsync) from live to the staging. This means all changes on staging will be overwritten somewhen. On staging the newest features and fixes can be tested using the current live data, before releasing them on the actual live system. Also the customer had access to this system to test and accept the new changes.

latest

On latest the HEAD of develop was checked out. All changes from multiple developers come together here on one system to be tested. This also means the system might be broken from time to time. Like on staging the database and assets (upload/ and fileadmin/) came from live, although a lot more rarely.

integration

Unfortunately we didn’t had a real integration system, where a developer/editor could prepare a new feature (create pages and content), test it and easily publish it to the live system (because content deployment is a not yet solved issue). We had to do this on staging (for testing and acceptance) and once again on live after releasing. For complex setups (multiple pages, a lot of content) we used .t3d exports to got that to the live system. From my personal experience you really need to be careful when ex-/importing t3d files!

Jenkins

We had a Jenkins, which took care of deploying code and data to the different system. These are Jenkins jobs we setup:

deploy to latest

This job was triggered on each commit (actually whenever someone pushed to the central git repository; using a git post-receive hook). It triggered an TYPO3 Surf deployment, which itself deploys the branch develop on latest and runs „EXT:coreapi“ on CLI to „clear caches“ and do a „DB compare“ on latest.

TYPO3 Surf was installed on the same machine Jenkins is running on. TYPO3 Surf and all Surf deployment configurations (for other projects too) were versionized in a separate Git repositories.

deploy to staging

This job wasn’t triggered automatically, but manually only. It also runs TYPO3 Surf to deploy the HEAD of release on staging, „clear the caches“ and do a „DB compare“.

deploy to live

This is almost the same as „deploy to staging“ except its deploys the head of master to live instead.

fetch live data to testing or staging

This job (manually triggered) did a mysqldump on the live system (excluding the caching and log tables) and applies it to the testing or staging system. All asset files (e.g. images) from live got synced using a rsync. The database was just a few hundred MBs and all asset files together sum up to like 5-6 GB (but most of them didn’t ever change, so rsync was pretty fast).

Whenever a developer felt like wanting to update his local system, he manually fetched a database dump from staging on the CLI and, if necessary, some asset files using rsync, too.

Deployment setups of others in the workshop

During the T3DD13 workshop we also discussed about setups and tools the participants are using:

Using Git seems to be very common these days, only a few are (still?) using SVN. When using Git, most people prefer using submodules to include other extensions or other repositories or just have „everything in one repository“. Using multiple branches (like master/staging/production or master/release/delevop) seems to be good practice.

Using composer instead seems to be a very promising alternative, but for TYPO3 CMS no one used that yet in production (in the time of asking). In the meanwhile Lightwerk/Felix Oertel setup a TYPO3 CMS composer repository containing all not-unsecure extensions, the TYPO3 CMS core and a custom composer installer. Using this repository one can setup a TYPO3 Installation using composer, instead of Git submodules.

Also Jenkins is very widely used to automate deployments. It triggers, depending on the personal preference, „TYPO.Surf“, „Ant“ or „phing“ to do the actual deployment.

I was very pleased to hear that EXT:coreapi found its place in many deployment setups to automatically clear caches or do Database scheme updates (DB compare). As an alternative (and maybe for historical reasons only ;-) some are using EXT:t3deploy and EXT:cleartypo3cache.