How to test a plugin before release?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

How to test a plugin before release?

Michael Clark-14
Last week I released an upgrade to a plugin. And started getting bug
reports that same day. Finally late last night I figured out what the
problem was. I and a few people test the new version, and was mostly
good. So this morning I released a new version, and am now getting
different bug reports. Argh! Of course, it's all working just fine on
my own sites.

So, how should I be testing my plugin? Searching for answers to this
question, I found this quote: "Free or not, any type of update
shouldn't be released until fully tested." There are a zillion
different themes and plugins out there, and there's no way to test for
every combination. Should I be building out a "beta test club" to help
me test new versions? I'm quite frustrated at the moment. And don't get
me started on the ever so helpful bug reports of "hey, it doesn't work
- fix it!" and "lame"

If you're curious, my plugin is at
http://wordpress.org/plugins/ultimate-category-excluder/ Mike


--

Michael Clark
http://www.PlanetMike.com

Christmas Music 24 Hours a Day, 7 Days a week
http://www.ChristmasMusic247.com

"Injustice anywhere is a threat to justice everywhere."
 - Martin Luther King Jr.
_______________________________________________
wp-hackers mailing list
[hidden email]
http://lists.automattic.com/mailman/listinfo/wp-hackers
Reply | Threaded
Open this post in threaded view
|

Re: How to test a plugin before release?

Bryan Petty
On Mon, Sep 9, 2013 at 3:08 PM, Michael Clark wrote:
> So, how should I be testing my plugin?

Stop looking at the problem from the perspective that you might need
to build a huge matrix of all sorts of versions of WordPress with
various popular pre-installed plugins combined with different versions
of PHP, and using different major settings like multisite. This is
still partially true, but it's not going to help you. Instead, there's
two major tactics that all plugin developers should be adopting that
mainly avoids this issue.

1. Use programming techniques that avoid conflicts and problems with
other plugins to begin with:

a. Prefix your global methods and classes (or use namespaces if using
OOP and requiring PHP 5.3+ with your plugin) and isolate 3rd party
libraries as a separate service layer if it frequently conflicts with
other common plugins also importing the same library (hide it behind a
separate web-requested PHP script that doesn't include WordPress if
possible).
b. Use your best judgement on ensuring all actions, hooks, and filters
you tie into have an appropriate priority level compared with how
other plugins use those same hooks if you need to use them, and always
make sure your filters return the exact same content if they don't
actually do anything with that content.
c. If your plugin writes HTML, CSS, or Javascript, do the same with
that: namespace and prefix JavaScript code, use unique prefixed CSS
classes on elements, avoid HTML element IDs, and use common built in
HTML element classes to pick up common theme styles.
d. If you want your plugin to work as far back as PHP 5.2 (and you
should since WP supports it), consider doing all of your development
in PHP 5.2 first, and then testing 5.4+ later - you can quickly get
used to the same old habits you used to have of *not* using
namespaces, closures, traits, short array syntax, etc.
e. I'm sure there's techniques others here could add to this list...

2. Add unit tests to your plugins, and think about configuring them
with a continuous integration server like Travis CI.

This will in fact build out a matrix of various configurations for you
covering multiple versions of WordPress, PHP, and multisite (and
common plugins if you customize it to), and run your unit tests across
*all* of them in a fully automated way that you can perform in as
little as a few minutes right before you tag a new release.

I've given a WordCamp presentation on this exact topic if it helps:
https://www.youtube.com/watch?v=9qMUc9anJKQ

Also see: https://github.com/tierra/wordpress-plugin-tests

When you have the hang of those two techniques, the only issues you
end up actually running into are mostly browser-specific bugs with
HTML/CSS. It's much harder to implement automated cross-browser
testing to locate and prevent these bugs before getting reports on
them, but they are usually minor anyway. You might still consider a
service like spoon.net/browsers , browsershots.org or browserstack.com
to help with that if you frequently run into those types of issues.

--
Regards,
Bryan Petty
_______________________________________________
wp-hackers mailing list
[hidden email]
http://lists.automattic.com/mailman/listinfo/wp-hackers
Reply | Threaded
Open this post in threaded view
|

Re: How to test a plugin before release?

Guus (IFS)
In reply to this post by Michael Clark-14
Hi Michael,

Good suggestion!

I have the same problem and the main issue is that I have no one to 'counter
test', like download the new version and install it 'fresh' or as an upgrade
'somewhere'.

So yeah, something like 'beta test' or 'counter test' would be very useful
for everybody. Let's think on how to organize,

Guus


----- Original Message -----
From: "Michael Clark"
<[hidden email]>
To: <[hidden email]>
Sent: Tuesday, September 10, 2013 5:08 AM
Subject: [wp-hackers] How to test a plugin before release?


> Last week I released an upgrade to a plugin. And started getting bug
> reports that same day. Finally late last night I figured out what the
> problem was. I and a few people test the new version, and was mostly
> good. So this morning I released a new version, and am now getting
> different bug reports. Argh! Of course, it's all working just fine on
> my own sites.
>
> So, how should I be testing my plugin? Searching for answers to this
> question, I found this quote: "Free or not, any type of update
> shouldn't be released until fully tested." There are a zillion
> different themes and plugins out there, and there's no way to test for
> every combination. Should I be building out a "beta test club" to help
> me test new versions? I'm quite frustrated at the moment. And don't get
> me started on the ever so helpful bug reports of "hey, it doesn't work
> - fix it!" and "lame"
>
> If you're curious, my plugin is at
> http://wordpress.org/plugins/ultimate-category-excluder/ Mike
>
>
> --
>
> Michael Clark
> http://www.PlanetMike.com
>
> Christmas Music 24 Hours a Day, 7 Days a week
> http://www.ChristmasMusic247.com
>
> "Injustice anywhere is a threat to justice everywhere."
> - Martin Luther King Jr.
> _______________________________________________
> wp-hackers mailing list
> [hidden email]
> http://lists.automattic.com/mailman/listinfo/wp-hackers 

_______________________________________________
wp-hackers mailing list
[hidden email]
http://lists.automattic.com/mailman/listinfo/wp-hackers
Reply | Threaded
Open this post in threaded view
|

Re: How to test a plugin before release?

Jaime Martínez
Hola,

If you want to setup unit tests then you could checkout WP-CLI to ease the initial setup. Here's a tutorial: http://wp-cli.org/blog/plugin-unit-tests.html

If you want to test fresh installs then checkout how WP-CLI runs it's Behat tests. With Behat you can do these "What happends when I have a fresh install".

Ciao,

Jaime Martinez



On 10 sep. 2013, at 06:01, "Guus \(IFS\)" <[hidden email]> wrote:

> Hi Michael,
>
> Good suggestion!
>
> I have the same problem and the main issue is that I have no one to 'counter test', like download the new version and install it 'fresh' or as an upgrade 'somewhere'.
>
> So yeah, something like 'beta test' or 'counter test' would be very useful for everybody. Let's think on how to organize,
>
> Guus
>
>
> ----- Original Message ----- From: "Michael Clark" <[hidden email]>
> To: <[hidden email]>
> Sent: Tuesday, September 10, 2013 5:08 AM
> Subject: [wp-hackers] How to test a plugin before release?
>
>
>> Last week I released an upgrade to a plugin. And started getting bug
>> reports that same day. Finally late last night I figured out what the
>> problem was. I and a few people test the new version, and was mostly
>> good. So this morning I released a new version, and am now getting
>> different bug reports. Argh! Of course, it's all working just fine on
>> my own sites.
>>
>> So, how should I be testing my plugin? Searching for answers to this
>> question, I found this quote: "Free or not, any type of update
>> shouldn't be released until fully tested." There are a zillion
>> different themes and plugins out there, and there's no way to test for
>> every combination. Should I be building out a "beta test club" to help
>> me test new versions? I'm quite frustrated at the moment. And don't get
>> me started on the ever so helpful bug reports of "hey, it doesn't work
>> - fix it!" and "lame"
>>
>> If you're curious, my plugin is at
>> http://wordpress.org/plugins/ultimate-category-excluder/ Mike
>>
>>
>> --
>>
>> Michael Clark
>> http://www.PlanetMike.com
>>
>> Christmas Music 24 Hours a Day, 7 Days a week
>> http://www.ChristmasMusic247.com
>>
>> "Injustice anywhere is a threat to justice everywhere."
>> - Martin Luther King Jr.
>> _______________________________________________
>> wp-hackers mailing list
>> [hidden email]
>> http://lists.automattic.com/mailman/listinfo/wp-hackers 
>
> _______________________________________________
> wp-hackers mailing list
> [hidden email]
> http://lists.automattic.com/mailman/listinfo/wp-hackers
_______________________________________________
wp-hackers mailing list
[hidden email]
http://lists.automattic.com/mailman/listinfo/wp-hackers