Archives for : tips and tricks

Part 2: Continuous Integration server to package Magento Extension with a click from JIRA

Posted by :Sven Varkel On : 07.07.2014
0

Category: tips and tricks, tutorials

Tags:, , , , , , ,

Starting Magento Connect extension packaging process from JIRA

When I wrote the first part of Continuous Integration server to package Magento Extension I didn’t know how to make it work with a click from JIRA. As you may know in JIRA there’s a way to release versions and trigger Bamboo builds during this release process. That’s what I was aiming – to make it work from JIRA so that it would NOT require any technical knowledge from let’s say Project Manager or Release Manager. I didn’t know it yesterday but here at MageFlow we have a learning mindset. I know it today:)

First – there are now 2 build plans configured in our Bamboo for our MageFlowConnect extension:

  1. “continuous builds” or “CI builds” that are triggered by commits that are made to the git repository that contains MFX code.
  2. “release builds” or “manual builds” that are triggered by clicking “Release” in JIRA

There are 2 main problems with creating Magento extension packages automatically:

  1. when and how to update extension’s version number (in module’s etc/config.xml and in package.xml mentioned in previous post)
  2. how to update release notes inside package.xml. However this one is not mandatory and it’s not resolved yet by us either.

Prerequisites

Install xsltproc to your build server and configure an executable “XSLTProc” in Bamboo.

Updating version numbers

In Part 1 I’ve created a XSL stylesheet that does the job of replacing extension version number in package.xml. Exactly the same XSL can be used to replace version number in extension’s etc/config.xml, too.

In your “CI build” process you may want to have extension version number increased with every build automatically. So that’s what we do  – we set the extension version number to that already defined in package.xml with added Bamboo build number. For example if package.xml contains version number 1.1.3 and Bamboo build # is 43 then the final version number of that extension build would be 1.1.3.43.

In order to replace version numbers in the “CI build” process add following 2 XSLTProc tasks to the build job in Bamboo:

  1. –output public/var/connect/package.xml –stringparam package_version ${bamboo.buildNumber} utils/update_version.xsl public/var/connect/package.xml
  2. –output public/app/code/community/Mageflow/Connect/etc/config.xml –stringparam package_version ${bamboo.buildNumber} utils/update_version.xsl public/app/code/community/Mageflow/Connect/etc/config.xml

First one appends ${bamboo.buildNumber} to existing version number (e.g 1.1.3) in package.xml while second does the same in etc/config.xml

In order to replace version numbers in the “Release build” (build triggered from JIRA) process add following 2 XSLTProc tasks to the build job in Bamboo:

  1. –output public/var/connect/package.xml –stringparam package_version ${bamboo.jira.version} utils/update_version.xsl public/var/connect/package.xml
  2. –output public/app/code/community/Mageflow/Connect/etc/config.xml –stringparam package_version ${bamboo.jira.version} utils/update_version.xsl public/app/code/community/Mageflow/Connect/etc/config.xml

First replaces version in package.xml with the version number specified in JIRA (let’s say 1.1.4) while second does the same in extension’s etc/config.xml

That’s it!:)

Please feel free to comment or send me questions directly.

 

 

Continuous Integration server to package Magento Extension

Posted by :Sven Varkel On : 02.07.2014
3

Category: general, tips and tricks, tutorials

Tags:, , , , , , , , , ,

Here’s the situation.

MageFlow is developing Magento Extension called MageFlow Connector. It’s an integration extension that is used to connect a Magento Instance to MageFlow. We are using Agile development processes and state-of-the-art tools by Atlassian like JIRA, Confluence, BitBucket, Bamboo and others. Bamboo is used as build and deployment server that automates a lot of development operations incl building and packaging our software.

Packaging Magento extensions – standard method

Usually Magento extensions are packaged in Magento backend. There’s a form that needs to be filled in with information about specific extension like version number, release notes, files and folders that should be packaged etc. After that you need to upload your extension to Magento Connect and publish it.

It’s pretty good tool but it requires manual work in order to create a package. However in our case we just despise any manual work:) Everything should be automated to max. That’s also why we even started creating MageFlow. Anyway …

Packaging Magento extensions with Bamboo CI

Packaging an extension with Bamboo requires some manual labor too. That cannot be avoided in my opinion. However …

Here’s my wild guess … With a really tight integration between JIRA and Bamboo and some tricks/scripts full automation could be achieved on the way from clicking “Create release” in JIRA to a fully packaged extension as an artifact of a Bamboo build. But right now it’s bit too much work for too little to gain.

But – keeping my head cold and not aiming 100% automation yet I created a following way to package an extension with Bamboo CI.

Prerequisites

GIT. For extension development we also use modman – a very useful tool for every Magento developer provided by Colin Mollenhour. It is also assumed that you have development environment ready in your local machine or elsewhere where you can prepare the package metainfo.

You also need to have a (preferred – vanilla) Magento (DevMagento) codebase in a GIT repository and you need to have YourExtension in a GIT repository. Make sure your DevMagento works fine and you can log in to the backend (admin).

Let’s name these GIT repositories like that for example:

DevMagento – git@bitbucket.com:myaccount/magento.git

MyExtension: git@bitbucket.com:myaccount/my_extension.git

Prepare Magento extension and modman

First you need to add MyExtension to DevMagento by using modman. In terminal do:

cd /srv/vhosts/magento
modman init
modman clone git@bitbucket.com:myaccount/my_extension.git

Second you need to create MyExtension XML metadata for Magento Connect package. This is done with the Magento Connect tool in Magento backend. Go to System->Magento Connect->Package extensions. Fill in all required fields there. Add version, release notes, files and folders, authors, dependencies etc. Click on “Save” button.

No back to Terminal:

cd /srv/vhosts/magento
ls public/var/connect

You see 2 very important files there now:

MyExtension.xml and package.xml

A mindnote here that package.xml will be overwritten if you package another extension in the backend of the same Magento while being in the same GIT branch.

Now create var/connect folder in your modmanned extension folder and move the files there:

cd /srv/vhosts/magento/.modman/my_extension
mkdir -p var/connect
mv ../../public/var/connect/MyExtension.xml ../../public/var/connect/package.xml var/connect/

Add these files to GIT and to your modman file.

Run modman update to re-create these files under Magento var/connect folder as symlinks:

cd /srv/vhosts/magento
modman update my_extension

Run a packaging test from command line to see if it works:

cd /srv/vhosts/magento/public
./mage package /srv/vhosts/magento/public/var/connect/package.xml
Done building package

It’s needless to say that you need to “git add”, “git commit” and “git push” your stuff etc. Now we’re almost ready to make Bamboo CI to create MyExtension-1.0.0.tgz. Also – see paragraph below about Magento bug after you finish next paragraph about configuring Bamboo.

Configure Bamboo CI

We are using standalone Bamboo CI in our own server. I’m not sure if everything looks the same in the ondemand solution…

Running unit tests on your Magento extension is not covered by this post. However I’m planning another post about this topic, too.

Install modman on your build server and add it as Bamboo executable if you haven’t done so yet.

In Bamboo create a new build plan called My Extension with plan key MYX. (Bamboo upper case plan keys and thus build paths with uppercase characters is where the Magento bug will strike in with its forced lowercase paths).

Add 2 repositories to your Build plan:

  1. DevMagento repository
  2. MyExtension repository

Make your build plan be triggered by periodically scanning the MyExtension repository for changes.

Configure build steps

Step 1: checkout source code from DevMagento repository. You may want to check out clean repo each time. It’s quite quick and you don’t build that extension too often. The reason is that otherwise next step will complain.

Step 2: run modman init. I.e choose modman executable and add “init” as command argument.

Step 3: run modman clone. Choose modman executable and add “clone git@bitbucket.org:myaccount/my_extension.git” as command argument

Step 4: Add a script as build step with ${bamboo.build.working.directory} as an argument to that command. The script is provided in this gist:

This script does 2 things:

  1. it runs mage script that creates Magento package
  2. it moves created tgz package from modmanned extension var/connect folder to Magento’s var/connect folder. The point is that Magento’s Package.php that creates the package uses realpath() PHP function. realpath() resolves symlinks created by modman and that’s why the .tgz package is created under .modman/my_extension/var/connect, not under public/var/connect.

After configuring build steps configure your build artifacts. Specify public/var/connect/ as artifact location and *.tgz as Copy pattern. You can make the artifacts shared so that a Bamboo Deployment plan can use the artifacts and deploy them to – let’s say – your test server;) But that’s another whole story …

Thanks!

Bug in Magento packaging scripts

NB! There’s a bug/misdesign in Magento packaging scripts. The solution is provided as a gist:

You may need to fix it in 2 places: under lib and under download. That’s why I recommended to use a vanilla Magento repository because you need to fix core library files. Create a special Magento repository, fix the bug there and use this Magento for packaging your extensions.

Alternatives

Alan Storm has created a good piece of software that enables creation of a Magento extension package from command line. However I found it too late and I’m not sure about 100% match with my problem.

Summary

It’s a pretty long post that covers quite complex issues. Feel free to leave a comment here or write me directly if you have any questions or thoughts about it.

 

Update: please see Part 2 of the same topic.

PHP system, exec, passthru and return code 127

Posted by :Sven Varkel On : 05.11.2013
0

Category: mindnotes, tips and tricks

Tags:, , , , , , , ,

This is a mindnote for myself. If someone else finds it useful, too == profit.

Because of MageFlow development I need to use quite a lot shell commands, for example ping.

Whenever exec, passthru or system returns code 127 check the path of the command. It means that the command cannot be found. It also means that PHP runs the command in such a shell that does not necessarily have PATH set properly and thus the command is not found.

Example

system('ping mygreathost.com', $retval);

$retval is 127.

Specify full path to ping command:

system('/bin/ping mygreathost.com', $retval);

$retval is 0 or 1 depending if mygreathost.com is available or not.

Oneliner for retrieving Magento database password from local.xml

Posted by :Sven Varkel On : 28.10.2013
0

Category: tips and tricks

Tags:, , , , ,

No comments:

How to create headless virtualbox machine with Debian Wheezy

Posted by :Sven Varkel On : 18.10.2013
0

Category: general, tips and tricks, tutorials

Tags:, , , , , ,

I’ve created a small helper script for those who want to be able to create VirtualBox virtual machine from command line via SSH and RDP.

Prequisites

As a prerequisite you need to download Debian’s network install CD image.

The script

Here’s the script and I’ll explain it below line by line:

How it works?

In short it works so that you run this script in your existing Linux and you give virtual machine name as the only parameter. It creates a new virtual machine with pretty safe values that all can be changed later (e.g amount of memory).

First we do initialization, check the scripts arguments and assign the first argument as VMNAME variable:

if test $# != 1; then
echo "Usage: $0 vm_name"
exit 0
fi
VMNAME=$1

Then we create and register new virtual machine of type 64bit Debian:

vboxmanage createvm --name $VMNAME --ostype Debian_64 --register

Then we allocate RAM, set power management, set DVD as first booting device, set network in bridged mode and we also specify that the machine would have 2 CPU-s with possibility to hotplug CPU-s:

vboxmanage modifyvm $VMNAME --memory 512 --acpi on --boot1 dvd --nic1 bridged --bridgeadapter1 eth0 --nictype1 virtio --cpus 2 --cpuhotplug on

After that hard drive image is created with size of 4 gigabytes. Make it larger if you have plenty of disk space handy.

vboxmanage createhd --filename ~/VirtualBox\ VMs/testvm/$VMNAME-disk01.vdi --size 4096 --variant Standard

Then we create 2 controllers, one for harddisks (SATA) and the other for CD/DVD (IDE):

vboxmanage storagectl $VMNAME --name "SATA controller" --add sata
vboxmanage storagectl $VMNAME --name "IDE controller" --add ide

Then we attach the newly created hard disk to the controller:

vboxmanage storageattach $VMNAME --storagectl "SATA controller" --port 0 --device 0 --type hdd --medium ~/VirtualBox\ VMs/$VMNAME/$VMNAME-disk01.vdi

As a last thing we attach the downloaded Debian 7 network installation CD image:

vboxmanage storageattach $VMNAME --storagectl "IDE controller" --port 1 --device 0 --type dvddrive --medium /mnt/raid/soft/debian-7.2.0-amd64-netinst.iso

… and we start the virtual machine:

vboxheadless -s $VMNAME &

You’ll be displayed a message similar to this one:

VRDE server is listening on port 3389.

Open a Remote Desktop Client and connect to your host machine’s IP and port 3389. You will see Debian’s installation screen. Install it and off you go!:)

Later you may want to set the default boot device to harddisk instead of DVD:

vboxmanage modifyvm $VMNAME --boot1 disk

Building GNUPG for PHP 5.5 on OSX

Posted by :Sven Varkel On : 02.10.2013
0

Category: tips and tricks, tutorials

Tags:, , , , ,

It’s quite tricky to build gnupg php extension on OSX. I’m using homebrew, so I installed necessary dependencies first:

brew install gpgme

This goes well as most things with brew.

Then I’m trying to install GNUPG PHP PECL extension as the instructions tell me to do:

sudo pecl install gnupg

Bang!!!

I get weird errors like these:

duplicate symbol _gnupg_keylistiterator_class_entry in:
.libs/gnupg.o
.libs/gnupg_keylistiterator.o
duplicate symbol _gnupg_class_entry in:
.libs/gnupg.o
.libs/gnupg_keylistiterator.o
ld: 2 duplicate symbols for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [gnupg.la] Error 1
ERROR: `make' failed

“Tough luck”, I think. But wait… After some googling around PHP bugs I got some hints.

So here’s what you got to do:

  1. download pecl extension source code
  2. fix 2 lines in .h and .c files
  3. phpize, configure and make
  4. configure PHP extension
  5. == Profit
pecl download gnupg

In my case it downloads files to /Users/sven/soft/php-5.5.4/ext/gnupg-1.3.3/gnupg-1.3.3.tgz

Unpack the archive:

tar xzf gnupg-1.3.3/gnupg-1.3.3.tgz
cd gnupg-1.3.3

Open file php_gnupg.h and add comments and explanation around line 50:

/* moved next line to gnupg.c
zend_class_entry *gnupg_class_entry;
 */

Copy the line zend_class_entry *gnupg_class_entry; to buffer. Save and exit.

Open file gnupg.c:

Find lines (around 177) and paste copied buffer so it looks like this:

zend_class_entry *gnupg_class_entry;
/* {{{ objects_new */
zend_object_value gnupg_obj_new(zend_class_entry *class_type TSRMLS_DC){

Do similar things with files gnupg_keylistiterator.c and php_gnupg_keylistiterator.h.

Open file  php_gnupg_keylistiterator.h around line 69:

/* moved next line to gnupg_keylistiterator.c
 zend_class_entry *gnupg_keylistiterator_class_entry;
 */

Copy line zend_class_entry *gnupg_keylistiterator_class_entry; to buffer. Save and exit.

Open file gnupg_keylistiterator.c around line 72 and make it look like this by pasting buffer:

zend_class_entry *gnupg_keylistiterator_class_entry;
/* {{{ keylistiterator_objects_new */
zend_object_value gnupg_keylistiterator_objects_new(zend_class_entry *class_type TSRMLS_DC){

Save and exit.

Configure, build and install: ./configure make clean && make make install

Now you should have php extension gnupg.so somewhere. I have it in my Cellar:

/usr/local/Cellar/php55/5.5.4/lib/php/extensions/no-debug-non-zts-20121212/gnupg.so

Wherever it is, copy the path to buffer.

Open a new file (your php-s conf.d may be somewhere else!)

vim /usr/local/etc/php/5.5/conf.d/ext-gnupg.ini

And add following lines there:

[gnupg]

extension="/usr/local/Cellar/php55/5.5.4/lib/php/extensions/no-debug-non-zts-20121212/gnupg.so"

Save and exit. Be sure to replace path if your gnupg.so is somewhere else.

Test your php:

php --ri gnupg

It should display something like this:

gnupg
gnupg support => enabled
GPGme Version => 1.4.3
Extension Version => 1.3.3-dev

Thanks!

 

 

 

Logging PHP errors in the same folder with Magento error logs

Posted by :Sven Varkel On : 14.08.2013
0

Category: general, tips and tricks

Tags:, , , ,

Here’s another post regarding logging…

How to log PHP errors to the same folder than Magento error logs?

It’s quite simple with Nginx and PHP5-FPM. In Nginx virtual host configuration file there is quite likely block like that:

location ~ .php$ {
    if (!-e $request_filename) { rewrite / /index.php last; }
    fastcgi_pass unix:/var/run/php5-fpm.sock;
    fastcgi_param   HTTPS $fastcgi_https;
    fastcgi_param   SCRIPT_FILENAME  $document_root$fastcgi_script_name;
    fastcgi_param   MAGE_RUN_CODE ee;
    fastcgi_param   MAGE_RUN_TYPE website;
    fastcgi_read_timeout    600;
    include /etc/nginx/fastcgi_params;
}

In order to make PHP error logs go to the same folder than Magento’s (/var/log/… add following lines to your virtual host configuration block:

    fastcgi_param   PHP_VALUE "log_errors=on";
    fastcgi_param   PHP_VALUE "display_errors=off";
    fastcgi_param   PHP_VALUE "error_log=$document_root/var/log/php_errors.log";

These are standard PHP directives.
log_errors – this one states that erros must be logged
display_errors – it’s always good to keep it off anywhere else than in your local development machine
error_log – this does the trick. It forces PHP logs for current virtual host to var/log folder under your Magento instance folder.

So the final block should look like this: 

location ~ .php$ {
    if (!-e $request_filename) { rewrite / /index.php last; }
    fastcgi_pass unix:/var/run/php5-fpm.sock;
    fastcgi_param   HTTPS $fastcgi_https;
    fastcgi_param   SCRIPT_FILENAME  $document_root$fastcgi_script_name;
    fastcgi_param   MAGE_RUN_CODE ee;
    fastcgi_param   MAGE_RUN_TYPE website;
    fastcgi_read_timeout    600;
    fastcgi_param   PHP_VALUE "log_errors=on";
    fastcgi_param   PHP_VALUE "display_errors=off";
    fastcgi_param   PHP_VALUE "error_log=$document_root/var/log/php_errors.log";
    include /etc/nginx/fastcgi_params;
}

Restart nginx and there you go. You can now tail or less the logs from /var/log/php_errors.log

Write better log when developing Magento extensions

Posted by :Sven Varkel On : 16.07.2013
0

Category: tips and tricks, tutorials

I’ve noticed that just a few developers add proper logging to their Magento extensions and libraries. Also – Magento core team is not part of those developers. Their logging habits are quite poor. I’m sorry, guys, but Magento is really bad when it comes to logging. Compare it to any J2EE project with proper Log4J set up and you’ll see the difference. Sometimes it’s good to learn from those old n’ bald Java guys, also:) OK, so much ranting this time… Now to the logging.

Is I said – Magento gives you tough times when it comes to debugging and searching for problems. Basically it just does not output any useful info to log files. It makes our – developers’ – life hard.

It is in our own hands to do it better because there are tools existing in Magento but these are regrettably underused.

What needs to be logged?

First – all problems and errors.
Whenever you use try/catch then please do write exception to log. Please do write all requests to API-s and responses. This would really help. There’s a special helper for logging exceptions: Mage::logException(Exception $e)
Use at least that one although it just spits randomly formatted Exception to var/log/exception.log. I recommend to use the solution below for exception logging also. Just change the Zend log priority to Zend_Log::CRIT or Zend_Log:ERR.

Second – debug log. When you develop anything then it’s really good to log output of some methods, input parameters that come from Ajax queries and so on. So it’s partly your little helper during development and it’s also part of good coding practices.

Here I have an advice for you about how to log. Do it like this and I’ll exlain why:

Mage::log(sprintf('%s(%s): %s', __METHOD__, __LINE__, print_r($var, true)), Zend_Log::DEBUG, 'mymodule.log');

Now I explain the format. After you have written a considerable amount of code you start noticing that you don’t need some debug lines any more. When you do logging without __METHOD__ and __LINE__ then you need to search thru all your source to find where the f*** did you add that line that outputs “OK” to log… And of course it just helps to discover problems quicker when you see __METHOD__ (which contains class (==file) information) and the line number in the code.
$var in print_r is your variable that you want to log. Be careful with large Magento objects! Because Magento object model is built so that it contains almost everything in every object and does that recursively then it would kill your PHP. Be warned!
Then there’s Zend_Log::DEBUG which is log priority level. Magento does not do smart logging that would consider log priority levels. But at least the log priority is displayed in the log file because Magento’s self log format is like that:
$format = '%timestamp% %priorityName% (%priority%): %message%' . PHP_EOL;

And the last parameter is file name. I support module-based logging. When I’m in a good mood I do logging to 2 files: my module’s log file and system.log. So the system.log would contain ALL logs and my module’s log file would contain logs from my module only.

Third – when you want to get really anal regarding all possible issues in your code then you want to log all notifications, too.

When in trouble then there’s never too much log. It’s really easy to set up log rotation in Linux or OSX systems so the large files aren’t a problem (now I gave you an idea for a good Magento extension;))