I am developing a parser in ruby which parses some nonuniform text data. Can anybody tell me, where I can get a good number of plaintext data for that?
Here's you'll get a list of many:
http://www.quora.com/Data/Where-can-I-get-large-datasets-open-to-the-public
And my fav is:
http://ftp.sunet.se/mirror/archive/ftp.sunet.se/pub/tv+movies/imdb/
You could scrape Wikipedia (or just run a bunch of it through lynx -dump). That would also give you a vast source of non-English text as well. Project Gutenberg would be another good source of large amounts of plain text.
Related
After testing URL in the google structured data testing tool, I don't know why the question mark character is shown instead of utf-8 characters?
what is wrong? any help really appreciated.
the url is : link
and the result image:
The underlying json is:
don't know why, but the structured data tool doesn't understand your farsi. If farsi is an encoded javascript, like \u0645\u0648\u0633\u0633\u0647 - there is no problem. But if it is written like علائم تیروئید کم کار - something wired happens.
Fast and dirty solution: encode all of your structured data content as encoded javascript. In Notepad++ this makes a plugin named HTML Tag - then ctrl+j.
I'm trying to OCR resumes. My first problem is, before OCR, to get the main blocks of a document.
Since all the resumes have "visual blocks" (referring to professional experience, skills, languages, hobbies, whatever ...), I wonder if there's any open source solution to "split" into "blocks" a document, obviously no matter the layout design (that's where some kind of AI will work, I assume)
Thank you
First decompress your pdf using zlib.
you will then be able to see the pdf in a readable format - https://web.archive.org/web/20141010035745/http://gnupdf.org/Introduction_to_PDF#A_first_example
The pdf format is kind of similar to postscript.
also try converting your pdf to postscript to see how contents are arranged.
you can decompress the pdf using pdf-parser https://blog.didierstevens.com/2008/10/30/pdf-parserpy/
try this as well - https://gist.github.com/averagesecurityguy/ba8d9ed3c59c1deffbd1390dafa5a3c2
Once you can see how your data is presented => you can then start applying alogorithms to extract more meaning.
I am building an Angular test preparation app (with Laravel 5.1 API). One of the requirements is to allow the user to print a certificate of achievement.
The client wants the person's name and credentials interpolated into the document (e.g., highlighted below). Here is a snapshot of the PDF template they sent:
The way I'm handling PDF viewing is simply by storing the file on S3 and giving them a link to that file.
Interpolating information into a PDF doc doesn't seem trivial and I haven't found much information on programmatically allowing this, but there are tools like DocHub, that allow you do edit while viewing the PDF.
I'm interested in learning:
is doing this programmatically trivial?
are there 3rd party tools I'm unaware of?
would I even be able to send this information along to the S3 link to interpolate in the first place?
Using PDF as a format for editing is usually a bad choice. If you have a form with fixed fields, then it's easy. Create a PDF template with an interactive form. In this form, based on AcroForm technology, you'll define fields with fixed coordinates, and a fixed size. You can then add content to these fields.
One major disadvantage with this approach is the lack of flexibility. Did you notice that I used the word "fixed" three times in the previous paragraph? If text doesn't fit the predefined field, you're out of luck. If the field is overdimensioned, you'll end up with plenty of white space. This approach is great if you can predict what the data will be like. A typical use case is a ticket or a voucher. For instance: the empty form is a really nice page, with only a couple of fields where an automated system can put a name, a date, a time, and a seat number.
This isn't the best approach for the example you show in your screen shot. The position of every line of text, every word, every character is known in advance. If you want to replace a short word with a long word (or vice-versa), then all those positions (of each line, of the complete page, possibly of the complete document) need to be recalculated. That's madness. Only people with very poor design skills come up with such an idea.
A better idea, is to store the template as HTML. See for instance chapter 5 of iText's pdfHTML tutorial, where we have this snippet of HTML:
<html>
<head>
<title>Invitation to SXSW 2018</title>
</head>
<body>
<u><b>Re: Invitation</b></u>
<br>
<p>Dear <name>SXSW visitor</name>,
we hope you had a great SXSW film festival experience last year.
And we would like to invite you to the next edition of SXSW Film
that takes place from March 9 until March 17, 2018.</p>
<p>Sincerely,<br>
The SXSW crew<br>
<date>August 4, 2017</date></p>
</body>
</html>
Actually, it's not really HTML, because the <name> tag and the <date> tag don't exist in HTML. All HTML processors (browsers as well as pdfHTML) ignore those tags and treat their content as if the tag was a <span>:
It doesn't make much sense to have such tags in the context of pure HTML, but it does make a lot of sense in the case of pdfHTML. With pdfHTMLL, you can configure custom tags, and have a result that looks like the PDFs shown below:
Look at the document for "John Doe" and compare it with the document for "Bruno Lowagie". The name "John Doe" is much shorter than my name, hence more words fit on that first line. The text flows nicely (we could also have chosen to justify the text on both sides). This "flow" is impossible to achieve with your approach, because you will never get a PDF template to reflow nicely.
OK, I get it, you probably say, but what about the practical aspects? You talk about a Java / .Net library, but I am working with Laravel and Angular.js. First, let me tell you that I don't think you'll find any good PDF tools for Laravel or Angular.js, because of the nature of PDF and those development environments (in my opinion, those technologies don't play well together). Regardless of my opinion, this shouldn't be much of a problem for you because you work in an Amazon environment. AWS supports Java, and the Java code needed to get pdfHTML working is minimal. Most of the code samples I wrote for the pdfHTML tutorial are shorter than 15 lines. So why not try Java and pdfHTML?
If you're already using Amazon services, why not use an amazon lambda function, in combination with iText7 (java), to generate the pdf on demand?
That way, you are guaranteed that the pdf is correct, and has nice layout every time.
Generating the pdf can either be done by:
converting HTML,
programmatically creating your entire document,
filling and flattening an XFA form.
I think for your use-case, either option 1 or 2 are the most sustainable.
I`m working at a little text editor. My application is a winapi one in C. The idea is to write text in a large textbox(like in notepad) and then when I press a button it will take all text into a buffer, format it after some rules and then put it in a .txt file.
For example, if my input is:
Anne \red(got) \blue(\bold(apples)) and \italic(\bold(snails!))
After I parse it, it`s possible to put it into a .txt file and after I open it to see it like this?
I want to thank everyone for their time. I got exactly what answer I wanted. Everyone here rocks
I think that you are programming for fun, just for the pleasure of it, and with the perspective of learning more. If that is the objective, then it is okay to invent your own formats and essay your own solutions.
The problem presented can be twofold:
does the format results need to be shown in the editor itself?
or do you just need to do something that is going to be rendered in an external program?
If you are after the first possibility, then you need some Win32 (given your environment) component that will show the formatting. That component is RichEdit, and it implements RTF, a codification that can be saved to a text file, and which is more or less standard.
If you have the second possibility in mind, then you can choose from a variety of codifications. You would just be creating a text editor, probably with some helpers that write part of the commands for the user. For example, you could be creating a HTML editor, or a RTF editor.
There is a third possibility, though. You create your own codification, and when saving, you translate that codification to HTML, and then open the document in a web browser.
Say that you have:
\bold(hello), world.
You would translate that to:
<html><body><b>hello</b>, world.</body></html>
The possibilities, as you can see, are inifinite.
Hope this helps.
new to stackoverflow so forgive me if I make any mistake.
I'm new to programming and scripting, although I have messed a little little bit with python and understand the basic of filemaker pro.
This is my problem: I have a full database that I built over the years. Database is just a way to say because, actually, it's a huge amount of rtf files with topics inside it. Now that I've built some real database I want to transfer my data from one to another. Just one table
The real problem is: in my old rtf days, I used to store my data in a easy to view manner, meaning that all my titles were bold/italic/underlined, and the text itself wasn't. So, I have aproximately 200 rtf files, each with 10-20 (sub)topics, waiting to be transfered to a two-columns table (title; content)
I would appreciate if anyone have a better idea than mine. My idea was to run a script that found the bold/italic text, copy it to my table field, find the not-bold, copy to the other field, etc. But I'm unable to find the answer to my simple question: how do I search for (and select) bold text?
I'd like to use applescript (is what I'm sort of comfortable with), but could use some other stuff.
You can try something like:
tell application "TextEdit"
set boldText to attribute runs of text of document 1 whose font contains "Bold"
end tell