I have a Drupal website that has a ton of data on it. However, people can quite easily scrape the site, due to the fact that Drupal class and IDs are pretty consistent.
Is there any way to "scramble" the code to make it harder to use something like PHP Simple HTML Dom Parser to scrape the site?
Are there other techniques that could make scraping the site a little harder?
Am I fighting a lost cause?
I am not sure if "scraping" is the official term, but I am referring to the process by which people write a script that "crawls" a website and parses sections of it in order to extract data and store it in their own database.
First I'd recommend you to google over web scraping anti-scrape. There you'll find some tools for fighting web scrapig.
As for the Drupal there should be some anti-scrape plugins avail (google over).
You might be interesting my categorized layout of anti-scrape techniques answer. It's for techy as well as non-tech users.
I am not sure but I think that it is quite easy to crawl a website where all contents are public, no matter if the IDs are sequential or not. You should take into account that if a human can read your Drupal site, a script also does.
Depending on your site's nature if you don't want your content to be indexed by others, you should consider setting registered-user access. Otherwise, I think you are fighting a lost cause.
I have a DotNetNuke 7 website that I have developed and an older version that was made in 4.9. I have to find a way to migrate the users into the new database so that the users of the website do not have to change their passwords or be inconvenienced in any fashion.
I have attempted to use plugins for this, but I cannot find one that exports 4.9 (seeing as 4.9 is so old) and another to import that file into 7.
If I try to go the manual migration route, what records would I need to pull over? I have noticed there are various user related tables. Are all of these mandatory for migrating users?
Thank you for your time. I'm just trying to understand how to go about the migration and would greatly appreciate some guidance.
You can do this via the database but you have a number of tables that you have to move across and you have Integer Based ID's that you have to worry about. Some of the tables to be aware of are aspnet_membership, aspnet_users, Users, UserProfile, ProfilePropertyDefinition, UserRoles, UserPortals and then there can be more as well.
DataSprings has a module called Interactive User Import which can also do an export. It works great and you can get a version that works that far back.
The other thing that you have to worry about if going truly the DB route is that passwords are encrypted, as such, if you move the passwords manually you have to keep the same encryption key.
I have a website made with CakePHP 1.3.7. This website has it's own login system. Now the client wants to include a forum in the website.
I've been looking at different free solutions and phpBB and SMF seem to be what I'm looking for. The only thing I'm not so sure is about integrating those forums with the login system that I already have.
I mean, if a user has already an account for the website (or creates a new one), he/she should be able to use that same account (username) in the forum section.
Is that possible? Any clue pointing me in the right direction would be much appreciated! I mentioned both forum solutions in case one is easier to integrate than the other one, that would be also good to know (or if there's any other better option).
Thanks so much in advance!
It's possible to use both but I personally prefer SMF. You have to configure CakePHP's session component to use database sessions and create a model that will use the forums session table.
You can decide if you want or need a separate users table besides the forums users table (or its called members, don't know right now).
The "hard" part is to make the cake app read/write the sessions and cookies in the same fashion SMF does to allow a smooth transition from the cake app to the forum and backwards.
Technically you can use both forums and archive your goal with both, it's just a matter of getting the frameworks components utilized right.
I ended up using: this
It has all that I needed and integrates perfectly into Cake :)
I am trying to making a tool which can help in maintaining data base version(like maintaining source code version). The technology which I am thinking to use is spring-hibernate so that the tool can be web based and it can be used by multiple project . The idea is that any database change can only be triggered with the help of this tool,so that the database version information can be maintained and the database can be made consistent .Operations like commit,roll back,branching,merging should be possible. Can you suggest me that how should I approach to this problem?
I have found an opensource tool called LiquidBase which has already provided some sort of solution in maintaining database version. Here is a short preview on what this tool can do. But this tool has some limitations like it does not handle stored procedures and triggers and it works on the basis of an XML file . But I think I can integrate this tool with my requirement and I can speed up development. If you have any other tool in knowledge which can be better than this then please let me know.
If possible tell me that how the tool should be organized so that different project can easily maintain their database version. What all problem the tool should try to address and what minimum support should at least be there in this tool? What should be the UI so that user should be easily able to use it.?
Is there a really good free tool for BugZilla reporting? I am finding the default search options on the web interface far too limiting. My biggest issue is with the lack of Order By options (only 1 field at a time, and a very limited set of fields to choose from). I have done some Google searches, but I can't find any good free BugZilla reporting tools.
If there isn't one, can someone please point me to an example on how to access the BugZilla web services? If I can get the BugZilla data, then I can easily build my own reports that will better meet our needs.
Take a look at this: http://www.faqs.org/docs/bugzilla/dbdoc.html
Use this database schema for reference: faqs.org/docs/bugzilla/dbschema.html
If you need a web-interface, use your favorite dynamic website scripting language that can access MySQL databases (say PHP)...
Simple-ish Tutorial: freewebmasterhelp.com/tutorials/phpmysql/4
PHP MySQL API Reference: php.net/manual/en/ref.mysql.php
Then use SQL queries such as:
"SELECT * FROM bugs WHERE WHERE bug_status != 'RESOLVED' ORDER BY creation_ts ASC, votes DESC LIMIT 50"
which lists first 50 entries of unresolved bugs ordered first ascending creation time then descending by number of votes.
I have used this in the past and have liked it a lot: http://www.mediawiki.org/wiki/Extension:Bugzilla_Reports
You can also consider other tool eg mantis
(http://www.mantisbt.org/)
I've personally switched from Bugzilla into Mantis and installed some plugins (http://deboutv.free.fr/mantis/) and found this more comfortable
If you are a Java user, you might want to check out Mylyn for eclipse. This is integrates a task-driven development approach into eclipse.
With that, you can raise bugs, tie together SVN changes and bugs, and hide classes that are not relevant to fixing bugs, etc. It's a bit involved to get started with, but quite powerful.
It also comes with a connector for BugZilla. See this introductory article for an example.
If you don't use eclipse, but you do use Java, then note that since Mylyn is open-source, you might want to look at the source code of the Mylyn BugZilla connector for how they do their work.
Good luck.
You can try Deskzilla (http://deskzilla.com/) - it is a multi-platform desktop client for Bugzilla with Outlook-like interface, rich reporting and filtering capabilities, offline work, drag-n-drop, etc. It's a commercial product, but if you're working on an Open Source project you can use it for free.
AFAIK Bugzilla uses MySQL database for storing data. So probably you can connect with some visual db manager (plenty of it exists, see Toad Data Modeler, DbVisualizer) and try do do some sql work...
There is a list of some add-ons (free and commercial) listed on the Buzilla addons wiki.
If you are a Windows user, MyZilla is a possible option.
Otherwise, to work toward your own, see the Bugzilla API documentation, which, in a way, includes how to retrieve the current schema (Bugzilla::DB::Schema), and Bugzilla::WebService.
Netbeans also has Bugzilla integration (I haven't tried it...).
I have analized a bunch of bug tracking tools.
You can try track or mantis, because bugzilla is very unfriendly about reporting.
Mantis
Mantis can export data in excel: all the graphic you need can be generated by that sheet.
For more information take a look to my blog:
http://gioorgi.com/2008/bug-tracking-mantis/
Anyway, Track is used a lot more, so for sake of completeness I should cite it:
Track
Pros:
Can Also work with an embedded database (using sqlite).
Easy to setup and use.
Cons:
Feature are too much, and aims to be also a CMS to some extend.
Take a look to:
http://gioorgi.com/2008/bug-tracking-trac/
Since Bugzilla can be installed on your own server, I presume the simplest way is to do that and play with the databases it creates ("Bugzilla supports MySQL, PostgreSQL and Oracle as database servers"). The documentation also says you can modify the templates as you like.
Otherwise one could try paid support or some other bug trackers.
I use this bookmarklet and like how it searches right with the strings entered in the location bar like smart search. It lets you quickly search bugzilla or jump to a bug number via Bugzilla Quicksearch, and is IE6+, Moz, Op7+ compatible.
Its companions on the same page can be used to refine or help with bug search/report, e.g. collect buglinks (queries bugzilla to show a list of bugs linked to from the current page),ord buglinkify (turns all numbers on the page into bug links).