What is the correct way to add an IPNS site to ipfs-desktop?
The purpose would be to seed a website on several computers, let's say 5.
So, I give my IPNS link to 5 of my friends. How can they contribute to hosting that site?
Related
I want to build a a webcrawler that goes randomly around the internet and puts broken (http statuscode 4xx) image links into a database.
So far I successfully build a scraper using the node packages request and cheerio. I understand the limitations are websites that dynamically create content, so I'm thinking to switch to puppeteer. Making this as fast as possible would be nice, but is not necessary as the server should run indefinetely.
My biggest question: Where do I start to crawl?
I want the crawler to find random webpages recursively, that likely have content and might have broken links. Can someone help to find a smart approach to this problem?
List of Domains
In general, the following services provide lists of domain names:
Alexa Top 1 Million: top-1m.csv.zip (free)
CSV file containing 1 million rows with the most visited websites according to Alexas algorithms
Verisign: Top-Level Domain Zone File Information (free IIRC)
You can ask Verisign directly via the linked page to give you their list of .com and .net domains. You have to fill out a form to request the data. If I recall correctly, the list is given free of charge for research purposes (maybe also for other reasons), but it might take several weeks until you get the approval.
whoisxmlapi.com: All Registered Domains (requires payment)
The company sells all kind of lists containing information regarding domain names, registrars, IPs, etc.
premiumdrops.com: Domain Zone lists (requires payment)
Similar to the previous one, you can get lists of different domain TLDs.
Crawling Approach
In general, I would assume that the older a website, the more likely it might be that it contains broken images (but that is already a bold assumption in itself). So, you could try to crawl older websites first if you use a list that contains the date when the domain was registered. In addition, you can speed up the crawling process by using multiple instances of puppeteer.
To give you a rough idea of the crawling speed: Let's say your server can crawl 5 websites per second (which requires 10-20 parallel browser instances assuming 2-4 seconds per page), you would need roughly two days for 1 million pages (1,000,000 / 5 / 60 / 60 / 24 = 2.3).
I don't know if that's what you're looking for, but this website renders a new random website whenever you click the New Random Website button, it might be useful if you could scrape it with puppeteer.
I recently had this question myself and was able to solve it with the help of this post. To clarify what other people have said previously, you can get lists of websites from various sources. Thomas Dondorf's suggestion to use Verisign's TLD zone file information is currently outdated, as I learned when I tried contacting them. Instead, you should look at ICANN's CZDNS. This website allows you to access TLD file information (by request) for any name, not just .com and .net, allowing you to potentially crawl more websites. In terms of crawling, as you said, Puppeteer would be a great choice.
I have 2 wordpress installs on one domain what I want to do is have both installs update posts as I add a post to the main install, how can I do this?
I believe it may be possible to have 2 front facing domains use the same database (share posts) but you will need to make the options parts different tables since they will use different themes/plugins I presume. Take a look at this page they seem to be on the right path. https://wordpress.org/support/topic/2-sites-1-db-possible?replies=15
You are facing a few problems.
First of all, wordpress saves domain names in the database. Also, in posts itself, when you add an image for example, the domain where you are on will be used in that post.
And as what you are doing is highly unusual, third party plugin builders don't take this situation into account, so any new plugin could break your setup in unexpected ways.
Another possible problem, but maybe not really one that concerns you is duplicate content. You'll get two exactly the same articles on two domains, without a canonical url.
Basically I would advise against this. There just are too many unknowns. Maybe a better approach is to investigate how to use WP multisite in this situation, and share content between the sites (this could do the trick), or use an alternative like ManageWP
My issue is that at my current job, we have to go to 7 different sites, depending on the carrier, to find out a part number through a model number. Each site allows you to search a database with a model number and if it is found it you can see the part number for it. This really helps us out, but I'm looking for a way to make it more convienient for us. Each site requires a login. How can I search all the sites, sites databases at once or how can I make something, webpage or program, that has a tab for each site and when clicked it takes me right to the search feature of the site. Any suggestions are appreciated. I really need to be pointed in the right direction. I almost forgot, if you know a site or program that does this for hvac part or appliances please let me know!
You won't be able to integrate directly with each site most likely, so you're stuck having to login to each and do your work.
But there is some hope!
Check out Selenium (http://docs.seleniumhq.org/) for automating web tasks. For example, you should be able to "record" yourself logging in and clicking to a page - and then in the future you just run a script and Selenium will do it for you. Should save a few steps.
You'd still need a tab open per site, but it should help.
i'm trying to find a good UAPROF website but only i can see that http://www.uaprof.com/ . I need UAPROFS of HTC and in that web a lot of links are broken. Do you know another web or resource to find that UAPROFS ?
Thank you in advance for your attention.
The Open Mobile Alliance validator keeps links to validated profiles. However the manufacturers do remove old profiles, or in some cases the manufacturers no longer exist, which is probably why you are seeing broken links at uaprof.com.
Google is another good source of profiles by searching for "uaprof filetype:xml" and "uaprof filetype:rdf".
Alternatively check out WURFL. This is not UAProf, but the data is derived from UAProf, it's XML so it is easier to process, and it is curated so it will not have the same problems.
There are more UAProf resources at the DELI website.
I want:
1. To be able to update multiple sites with one upload
2. News page will now database driven
3. Updates in common elements likes LOGOS, COUNTRIES, ADDRESSES, TELEPHONES, FAX NUMBER and EMAIL ADDRESSES can be done with limited HTML knowledge
Hello guys..Is this possible in Drupal 7?
Thanks
It is possible.
But you might need to search for right module or write your own. You might need install the features module or the domain access. But what you are asking is quite general question, with Drupal everything is possible. But you might need to write PHP code for matching your special needs.