I just committed some changes to feedstock that make it require Ruby (and the standalone migrations gem). The biggest question is probably why have a project that is written in php now require Ruby. There are a few reasons for this so I should probably get started explaining them.
I have a friend who works with Ruby on a daily basis and he was showing me active record migrations and how simple they were. I was pretty excited as to how simple they were and spent quite a bit of time looking for something that was the same in php. I found a few projects but didn't feel like they suited feedstock that well. All I really wanted was the migrations as I felt it would streamline development for feedstock. I would be free to make changes to the database when I needed and would not have to worry about what version the user was on. I was left with two options, write my own migration program in php or use Ruby and the standalone migrations gem. The standalone migrations gem fit what I was looking for so I decided to use it. It also makes it a lot easier to support various different types of databases.
A second reason is that a few people I know have wanted me to learn Ruby, and subsequently Rails. This seemed like a good reason to add some Ruby to feedstock as I would learn a tiny bit about the language and see how things work. I could get into some of the things I find a bit funny, but thats probably because im coming from the java/c/php world.
I wanted to address the long time from the last feedstock alpha until now. Alpha 4 is still not done and I don't have any definite date set. I've worked on quite a few things in between the last alpha and now (various feedstock things and other projects) which you can find on my github page.
I have a friend who works with Ruby on a daily basis and he was showing me active record migrations and how simple they were. I was pretty excited as to how simple they were and spent quite a bit of time looking for something that was the same in php. I found a few projects but didn't feel like they suited feedstock that well. All I really wanted was the migrations as I felt it would streamline development for feedstock. I would be free to make changes to the database when I needed and would not have to worry about what version the user was on. I was left with two options, write my own migration program in php or use Ruby and the standalone migrations gem. The standalone migrations gem fit what I was looking for so I decided to use it. It also makes it a lot easier to support various different types of databases.
A second reason is that a few people I know have wanted me to learn Ruby, and subsequently Rails. This seemed like a good reason to add some Ruby to feedstock as I would learn a tiny bit about the language and see how things work. I could get into some of the things I find a bit funny, but thats probably because im coming from the java/c/php world.
I wanted to address the long time from the last feedstock alpha until now. Alpha 4 is still not done and I don't have any definite date set. I've worked on quite a few things in between the last alpha and now (various feedstock things and other projects) which you can find on my github page.
Categories: projects
Date: 02/03/12
One of the many changes in the feedstock alpha 3 is the new way caching works. Before there were two different types of caching, static and dynamic. Each of those needed their own classes to communicate with the backends, Ex. ApcDynamic and ApcStatic. There are still static and dynamic options but they don't need separate classes to run. The static option stores the final output of a page so when it is called again it just loads the final output and sends it to the client. The dynamic option caches sql queries and their results, this way pages are still generated on each page load.
I was able to get rid of two separate cache backends by creating an interface called GenericCacher and creating a CacherCreator class. The GenericCacher interface defines functions that each cacher must have. The CacherCreator class checks to see if the currently loaded cacher is an instance of the GenericCacher interface. If it is an instance of GenericCacher then we know it is going to implement all of the functions in GenericCacher. By requiring each cacher implement GenericCacher I was able to structure my code so both forms of caching, static and dynamic, are supported by the same cacher class.
Now FileCache supports both static and dynamic caching, where it previously only supported static caching. This means that if a host doesn't support anything like APC or Xcache someone could cache either sql or fully generated pages on disk. I ended up having FileCache serialize and unserialize data when writing and reading files from disk. The nice thing with serialize is I can store arrays in files and read them back into an array with the same format, definitely comes in handy when the sql results are going to be arrays. Using FileCache is usually slower than using APC or Xcache (This is because those caching backends store data in memory) but it is usually faster than running without any caching.
One of the nice things about rewriting the way caching works is that it is now really easy to add support for new caching methods. One of the caching methods feedstock currently does not support is Memcached. The main reason for this is that I don't have a Memcached server to test things on. If I were to support Memcached it wouldn't be hard to get it added to feedstock. All I would need to do is add a Memcached.php file (and a Memcached class inside of that) that implements GenericCacher. Once there I would just need to set up the logic and then change $this->config['cacheName'] = 'something' to = 'Memcached' . There will probably need to be some changes to the way caching classes work because Memcached would need to take in a list servers it can connect to.
Something I might add to GenericCacher is allowing the constructor take in an optional array of options at the end. This would allow future caching methods (like Memcached) to take in custom options. These custom options could be a list of servers to connect to.
To sum things up the way caching is implemented in feedstock alpha 3 is greatly improved over previous versions. It is now easier to support new caching methods, easier to make changes to the way caching methods work (via things like GenericCacher interface), and easier to support the two (static/dynamic) types of caching data.
I was able to get rid of two separate cache backends by creating an interface called GenericCacher and creating a CacherCreator class. The GenericCacher interface defines functions that each cacher must have. The CacherCreator class checks to see if the currently loaded cacher is an instance of the GenericCacher interface. If it is an instance of GenericCacher then we know it is going to implement all of the functions in GenericCacher. By requiring each cacher implement GenericCacher I was able to structure my code so both forms of caching, static and dynamic, are supported by the same cacher class.
Now FileCache supports both static and dynamic caching, where it previously only supported static caching. This means that if a host doesn't support anything like APC or Xcache someone could cache either sql or fully generated pages on disk. I ended up having FileCache serialize and unserialize data when writing and reading files from disk. The nice thing with serialize is I can store arrays in files and read them back into an array with the same format, definitely comes in handy when the sql results are going to be arrays. Using FileCache is usually slower than using APC or Xcache (This is because those caching backends store data in memory) but it is usually faster than running without any caching.
One of the nice things about rewriting the way caching works is that it is now really easy to add support for new caching methods. One of the caching methods feedstock currently does not support is Memcached. The main reason for this is that I don't have a Memcached server to test things on. If I were to support Memcached it wouldn't be hard to get it added to feedstock. All I would need to do is add a Memcached.php file (and a Memcached class inside of that) that implements GenericCacher. Once there I would just need to set up the logic and then change $this->config['cacheName'] = 'something' to = 'Memcached' . There will probably need to be some changes to the way caching classes work because Memcached would need to take in a list servers it can connect to.
Something I might add to GenericCacher is allowing the constructor take in an optional array of options at the end. This would allow future caching methods (like Memcached) to take in custom options. These custom options could be a list of servers to connect to.
To sum things up the way caching is implemented in feedstock alpha 3 is greatly improved over previous versions. It is now easier to support new caching methods, easier to make changes to the way caching methods work (via things like GenericCacher interface), and easier to support the two (static/dynamic) types of caching data.
Categories: projects
Date: 04/14/11
After a long wait the third alpha of feedstock is ready!
You can find it on my github repo. There have been a lot of changes to the front-end of feedstock in this release. As far as changes go most are going to be transparent to users, minus the new config format (there is a config.sample.php on github you can use to convert the old config to the new format). I will hopefully have some time over the next few weeks to go into more detail the changes that took place.
Some quick stats from Alpha 2 to Alpha 3 there were 50+ files with changes 3,400+ additions and 1,900 deletions.
I hope you enjoy it.
You can find it on my github repo. There have been a lot of changes to the front-end of feedstock in this release. As far as changes go most are going to be transparent to users, minus the new config format (there is a config.sample.php on github you can use to convert the old config to the new format). I will hopefully have some time over the next few weeks to go into more detail the changes that took place.
Some quick stats from Alpha 2 to Alpha 3 there were 50+ files with changes 3,400+ additions and 1,900 deletions.
I hope you enjoy it.
Tags: Feedstock
Categories: projects
Date: 03/25/11
I haven't been able to spend a lot of time working on the awesome feedstock, partly because after I get home from work I don't want to do any more programing. I've made a lot of little changes over the past month or so but here are some of the bigger ones.
APC support. I figured that since APC is going to be bundled with php 6 I should support it, plus it's really good and we are currently running it here.
Lazy database connecting. So if all the needed DB queries are already cached we don't even spend time connecting to the database. The nice thing is that performance should increase but we don't know if the database is down until we actually need to get something from it. I have to possibly work out better errors for it, it currently uses trigger_error().
Lastly sitemap generation. Sitemap's are generated whenever there is a new post/page/category. They come in handy for web crawlers so they know every page that feedstock can show. Sometime in the future I would like to support siteindex's, they allow multiple sitemap's (sitemaps can only hold something like 50000 links).
Like I said before there was a bunch of behind the scenes changes and some mysql query changes (now that I understand JOINS). If you find any bugs feel free to contact me, you can find that info in the navigation right up ^^ there.
Oh and I should mention http://redmine.niftystopwatch.org has been taken down, you can find out why if you go there.
APC support. I figured that since APC is going to be bundled with php 6 I should support it, plus it's really good and we are currently running it here.
Lazy database connecting. So if all the needed DB queries are already cached we don't even spend time connecting to the database. The nice thing is that performance should increase but we don't know if the database is down until we actually need to get something from it. I have to possibly work out better errors for it, it currently uses trigger_error().
Lastly sitemap generation. Sitemap's are generated whenever there is a new post/page/category. They come in handy for web crawlers so they know every page that feedstock can show. Sometime in the future I would like to support siteindex's, they allow multiple sitemap's (sitemaps can only hold something like 50000 links).
Like I said before there was a bunch of behind the scenes changes and some mysql query changes (now that I understand JOINS). If you find any bugs feel free to contact me, you can find that info in the navigation right up ^^ there.
Oh and I should mention http://redmine.niftystopwatch.org has been taken down, you can find out why if you go there.
Tags: Feedstock
Categories: projects
Date: 07/26/10
Before I start this is more about plugins for web apps than desktop apps. I could get into the differences between them but I'm going to hope you have the same views I do on the differences.
When I started working on feedstock one of the decisions I made was to not support plugins. It's not that I dislike plugins, because they can be really great. It's just that anyone can write them and it seems that the more you add the slower things get. Since feedstock is written in php and php isn't a compiled language (I understand there are things like phpc and hiphop that can compile php). So lets say someone comes to visit the site, I load all my "core" stuff then check for plugins and load those, then start running. If the plugins were to register onto things when when I get to that point we need to run stuff in the plugin (the plugin could have really bad time complexity O(2^n) would be bad but probably wouldn't happen) then go back to doing what we were previously doing. So for every plugin that is added it needs to be loaded and then wherever it registers we need to run it.
On top of that I would need to provide an interface to allow plugins to interact with the core of feedstock. I am not at a point where I feel like I could handle this. Then I would have to deal with possibly breaking things when feedstock was updated. When it's just me as the developer I would rather spend my time making cool things and fixing bugs (not that fixing bugs is super fun but hey things need to work).
In the end I would rather spend my time making the core functionality of feedstock pretty good, since I don't have an unlimited amount of time.
When I started working on feedstock one of the decisions I made was to not support plugins. It's not that I dislike plugins, because they can be really great. It's just that anyone can write them and it seems that the more you add the slower things get. Since feedstock is written in php and php isn't a compiled language (I understand there are things like phpc and hiphop that can compile php). So lets say someone comes to visit the site, I load all my "core" stuff then check for plugins and load those, then start running. If the plugins were to register onto things when when I get to that point we need to run stuff in the plugin (the plugin could have really bad time complexity O(2^n) would be bad but probably wouldn't happen) then go back to doing what we were previously doing. So for every plugin that is added it needs to be loaded and then wherever it registers we need to run it.
On top of that I would need to provide an interface to allow plugins to interact with the core of feedstock. I am not at a point where I feel like I could handle this. Then I would have to deal with possibly breaking things when feedstock was updated. When it's just me as the developer I would rather spend my time making cool things and fixing bugs (not that fixing bugs is super fun but hey things need to work).
In the end I would rather spend my time making the core functionality of feedstock pretty good, since I don't have an unlimited amount of time.
Tags: Feedstock
Categories: projects
Date: 06/29/10
Now that school is out for summer I have a bit of time to spend on feedstock. I am currently at the point where other people besides me testing stuff would be great. You can even file bug reports and feature requests on http://redmine.niftystopwatch.org(sorry if it takes some time to load). I would really enjoy your feedback on feedstock. Oh and feel free to contact me if you have any questions with it.
Tags: Feedstock
Categories: projects
Date: 05/21/10
I figured this would come up at some point in time and I wanted to address it. Why doesn't Feedstock support commenting on posts? It has nothing to do with a technical limitation. It's not something I could implement in a day as it would take some time. I could also come up with a way to disable comments for a site/posts and not affect performance when disabled. I feel as though there are already good commenting systems out there. Disqus and Intense Debate to name a few. Those should be pretty simple to implement as it seems like they don't require any core modifications to feedstock code, they can be implemented at the theme level.
Another reason would be having to monitor commenting for spam. There are systems out there that help with that, akismet and Typepad AntiSpam. Even when I ran a small traffic blog spam still got through, and the amount of spam that was caught was pretty insane.
So maybe at some time Feedstock will support it's own commenting but currently there are reasons for me to focus development in other areas.
Another reason would be having to monitor commenting for spam. There are systems out there that help with that, akismet and Typepad AntiSpam. Even when I ran a small traffic blog spam still got through, and the amount of spam that was caught was pretty insane.
So maybe at some time Feedstock will support it's own commenting but currently there are reasons for me to focus development in other areas.
Tags: Feedstock
Categories: projects
Date: 05/08/10
One of the things I've wanted to do with Feedstock since I started on it was, at some point, to support PubSubHubBub. Today In class I was checking out how to get publishing working and it turns out it is pretty easy. When I got home I sat down and started working with it and ended up writing my own PubSubHubBub php class, ok well I could make it more generic but still it works!
Since there's two feeds that Feedstock can handle I need to send two separate notifications to the hub we are using (you can only use one hub write now, I *think* it is possible to have more than one hub). I opted to use multi_curl which hopefully saves some posting time as it sends both posts at the same time. Both the ATOM and the RSS2.0 feeds support PubSubHubBub in Feedstock. Going to http://matthewshafer.net/feed/ defaults to the RSS2.0 feed whereas http://matthewshafer.net/feed/atom/ goes to the ATOM feed. Anyway if anyone is having issues with the feeds you can always send me an e-mail.
Since there's two feeds that Feedstock can handle I need to send two separate notifications to the hub we are using (you can only use one hub write now, I *think* it is possible to have more than one hub). I opted to use multi_curl which hopefully saves some posting time as it sends both posts at the same time. Both the ATOM and the RSS2.0 feeds support PubSubHubBub in Feedstock. Going to http://matthewshafer.net/feed/ defaults to the RSS2.0 feed whereas http://matthewshafer.net/feed/atom/ goes to the ATOM feed. Anyway if anyone is having issues with the feeds you can always send me an e-mail.
Tags: Feedstock
Categories: projects
Date: 04/30/10
In my post yesterday I mentioned that I was managing what data I had written to xcache in my xcacheDynamic class because if a post is added while there are cached values we need to clear those values so users can see the new post right away.
In the xcacheStatic class I tried to use xcache_unset_by_prefix() but I assume that is a xcache 1.3 feature as in 1.2.2, which I currently run, it caused a function not defined error. This definitely makes things harder as I attach a prefix to every entry in xcache so deleting using this function makes things trivial.
This is what I did in the xcacheStatic and xcacheDynamic classes, I used an associative array. So essentially what that means is if the prefix is test_ then in the cache there is a test_array entry that stores an associative array containing all the items I have added to the cache. Why an associative array? I can set the key to the hash of the query and the value to the hash of the query. Then it is stored inside the var cache. This makes it easy to see if the value already exists in the array using isset(), thus if it exists I don't need to push something else onto the array. It's also nice because when we purge the cache I am able to iterate through the array and each value is the hash i need to delete. The nice thing in PHP is you can iterate through associative arrays.
The only reason I need this is if an update/insert/delete happens to something in the database and the cache hasn't been garbage collected yet. This way theres no waiting for the cache to expire in order to show new data, it is shown after the data is inserted in the database. Now I wouldn't need to use the array method if I were able to delete by prefix, but thats okay as this method works fine.
Oh and I only need to update the array when something is being written to the var cache.
the pseudocode is something like
if(don't have the array in xcache)
{
create an array containing the hash as the key and as the value.
write that as something like prefix_somethingweknow into xcache
}
else
{
grab the array from xcache
check if it contains this value
if it doesn't then add it and write the entire thing to xcache
}
In the xcacheStatic class I tried to use xcache_unset_by_prefix() but I assume that is a xcache 1.3 feature as in 1.2.2, which I currently run, it caused a function not defined error. This definitely makes things harder as I attach a prefix to every entry in xcache so deleting using this function makes things trivial.
This is what I did in the xcacheStatic and xcacheDynamic classes, I used an associative array. So essentially what that means is if the prefix is test_ then in the cache there is a test_array entry that stores an associative array containing all the items I have added to the cache. Why an associative array? I can set the key to the hash of the query and the value to the hash of the query. Then it is stored inside the var cache. This makes it easy to see if the value already exists in the array using isset(), thus if it exists I don't need to push something else onto the array. It's also nice because when we purge the cache I am able to iterate through the array and each value is the hash i need to delete. The nice thing in PHP is you can iterate through associative arrays.
The only reason I need this is if an update/insert/delete happens to something in the database and the cache hasn't been garbage collected yet. This way theres no waiting for the cache to expire in order to show new data, it is shown after the data is inserted in the database. Now I wouldn't need to use the array method if I were able to delete by prefix, but thats okay as this method works fine.
Oh and I only need to update the array when something is being written to the var cache.
the pseudocode is something like
if(don't have the array in xcache)
{
create an array containing the hash as the key and as the value.
write that as something like prefix_somethingweknow into xcache
}
else
{
grab the array from xcache
check if it contains this value
if it doesn't then add it and write the entire thing to xcache
}
Categories: projects
Date: 04/29/10
A few days ago I was reading about caching SQL queries using things like memcached. So I decided to test out the possibility of adding it to the database queries. I wrote an xcacheDynamic class that handles keeping track of the cached var's. More on why I have to manually keep track of the var's cached by xcache in another post. I was seeing pretty decent performance increases on my local build but one thing to note is that when I run MySQL queries the mds program on my mac decides it needs to index stuff so that uses a ton of processor. Anyway even doubling the requests a second from no caching was still less than with caching. The thing is that I have Indexes set up in MySQL which should make the SELECT's have pretty good performance. I do think that on large datasets the caching will probably show a bigger advantage when it comes to performance. There are probably some places where I could cut corners with caching make things quick but cutting corners can lead to issues. But I did notice some places where I could improve the code which might lead to slightly higher performance.
I still need to run more performance tests, possibly I will run some tests in a VM once I get the code cleaned up. The nice thing with this way of caching is that themes can still be dynamic, so themes can kill performance but at least the "core" of Feedstock is quick.
I still need to run more performance tests, possibly I will run some tests in a VM once I get the code cleaned up. The nice thing with this way of caching is that themes can still be dynamic, so themes can kill performance but at least the "core" of Feedstock is quick.
Tags: Feedstock, performance
Categories: projects
Date: 04/28/10
Next Page ->