I have a c# application which reads a table of roughly 1500 site url's of clients who have been with the company since we started. Basically, I am running whois queries on these url's and seeing if they are still a client or not. The application works but it takes roughly an hour to complete. Would I be better off using async whois queries and how much time roughly could I save.
Here is a sample whois query block of code that I am using.
Also if anyone has any tips on how to improve this code or run async commands could ye please help me out as I'm only an intern. Thanks
string whoisServer = "whois.markmonitor.com";
string data;
try
{
TcpClient objTCPC = new TcpClient(whoisServer, 43);
string strDomain = domainName + "\r\n";
byte[] arrDomain = Encoding.ASCII.GetBytes(strDomain);
Stream objStream = objTCPC.GetStream();
objStream.Write(arrDomain, 0, strDomain.Length);
StreamReader objSR = new StreamReader(objTCPC.GetStream(),
Encoding.ASCII);
//return objSR.ReadLine();
//return (Regex.Replace(objSR.ReadToEnd(),"\n","<br>")).ToString();
using (StreamReader reader = new StreamReader(objTCPC.GetStream(), Encoding.ASCII))
{
data = (reader.ReadToEnd());
}
//test.Add(objSR.ReadLine());
objTCPC.Close();
}
catch
{
data = "Not Found";
}
return data;
Well, the short answer is certainly yes.
Since you are making multiple, completely independent lookups, you have everything to gain by running them in parallel, asynchronously.
There are several ways to do this. The options depend on what version of .net you're in.
As you would guess, there are many examples.
Check these out right here on SO.
Avaliable parallel technologies in .Net
Multi threaded file processing with .NET
When to use a Parallel.ForEach loop instead of a regular foreach?
Related
My problem statement goes like this
" I want to leverage apache solr 8.6.1 streaming capability to send millions of records as part of spring boot REST API call. I cannot directly call solr end points due to security restrictions and also some business logic in place. So I have written the code through which I am able to read the data as stream and push it to spring boot outputstream."
When I am making the API call everytime It goes through the following code
StreamFactory factory = new StreamFactory().withCollectionZkHost(COLLECTION_NAME,ZK_HOST);
SolrClientCache solrClientCache = new SolrClientCache(httpClient);
StreamContext streamContext = new StreamContext();
streamContext.setSolrClientCache(solrClientCache);
String expressionStr = String.format(SEARCH_EXPRESSION,COLLECTION_NAME);
StreamExpression expression = StreamExpressionParser.parse(expressionStr);
TupleStream stream;
try {
stream = new CloudSolrStream(expression, factory);
stream.setStreamContext(streamContext);
stream.open();
Tuple tuple = stream.read();
int count = 0;
while (!tuple.EOF) {
String jsonStr = ++count + " " + tuple.jsonStr() + "\r\n";
outputStream.write(jsonStr.getBytes());
outputStream.flush();
tuple = stream.read();
}
stream.close();
} catch (IOException e) {
e.printStackTrace();
}
and it tries to connect to zookeeper at stream.open and it is taking some time.
is it possible to optimize this code so that everytime it doesn't have to connect to zookeeper and we can keep it ready before hand only.
because it is a stream that's why we have to open and close the stream with every call.
also how it will behave in the multiuser scenario.
Can anyone throw some light on it and how we can optimize it further
Sorry, If u filling bored. I have searched on several search engines but could not got any result. Anyway I am working in an App which database is mysql. Now I have created a database wrapper class and want to check if the connection is already opened. Could u help me?
String^ constring = L"datasource=localhost;port=3306;username=root;password=pass;database=eps;";
String^ my_query = L"select id from eps_users where usr = '" + this->user_name->Text + "' and psw = md5('" + this->pass_word->Text + "');";
MySqlConnection^ conDatabase = gcnew MySqlConnection(constring);
MySqlCommand^ cmd = gcnew MySqlCommand(my_query, conDatabase);
MySqlDataReader^ myreader;
try
{
conDatabase->Open();
myreader = cmd->ExecuteReader();
int count = 0;
while (myreader->Read())
{
count = count + 1;
}
if (count == 1){
MessageBox::Show("Username And Password is correct.", "Success", MessageBoxButtons::OK,
MessageBoxIcon::Information);
this->Hide();
Form2^ f2 = gcnew Form2(constring);
f2->ShowDialog();
}
else{
MessageBox::Show("Username And Password is not correct.", "Error", MessageBoxButtons::OK,
MessageBoxIcon::Error);
// <del>
this->Hide();
Form2^ f2 = gcnew Form2(constring);
f2->ShowDialog();
// </del>
}
}
catch (Exception^ ex)
{
MessageBox::Show(ex->Message);
}
conDatabase->Close();
I need to check if( conDatabase->HasBeenOpened()) { conDatabase->Open();}
The MySqlConnection type implements a feature called connection pooling that relies on the garbage collector to help recycle connections to your database, such that the best practice with regards to connection objects is to create a brand new object for most calls to the database, so that the garbage collector can correctly recycle the old ones. The process goes like this:
Create a new connection
Open the connection
Use the connection for one query/transaction
Dispose the connection
Where all four steps live within a single try/catch/finally block. (Also, the dispose step needs to happen inside the finally block!) Because you generally start with a brand new connection object, there's not typically a need to check if it's open first: you know it's closed. You also don't need to check the state after calling Open(): the method will block until it's finished, and throw an exception if it fails.
However, if you really are in one of the (rare!) situations where it's a good idea to preserve the connection for an extended period, you can check the state like this:
if( conDatabase->State == ConnectionState::Open)
Now, there is one other issue in that code I'd like to talk about. The issue comes down to this: what do you think will happen if I put the following into your username text box:
';DROP Table eps_users;--
If you think that it will try to execute that DROP statement in your database, you're right: it will! More subtle and damaging queries are possible, as well. This is a huge issue: there are bots that run full time crawling web sites looking for ways to abuse this, and even an corporate internal desktop apps will get caught from time to time. To fix this, you need to use Parameterized Queries for every instance where include user-provided data as part of your sql statement.
A quick example might look like this:
String^ my_query = L"select id from eps_users where usr = #userID;";
MySqlCommand^ cmd = gcnew MySqlCommand(my_query, conDatabase);
cmd->Parameters->AddWithValue(L"#userID", this->user_name->Text);
In SL4 application I perform a lot of requests (30) to the server in the same time (+~3 sec) by using HttpWebRequest. As a result I receive the same count of responses and start to process streams
HttpWebRequest request = (HttpWebRequest) asynchronousResult.AsyncState;
HttpWebResponse response = (HttpWebResponse) request.EndGetResponse(asynchronousResult)
using (var stream = new StreamReader(response.GetResponseStream()))
{
var str = stream.ReadToEnd();
var doc = XDocument.Parse(str);
}
All responsies are received (server returned 304 - not modified)
For each response program creates new thread (because of asynchronusResult).
When I try to fetch data from stream it takes a lot of time when stream.ReadToEnd() method works. I don't now what inside ReadToEnd() but it seems that stream reading performs in one thread or any locking occurs. Any ideas??
Why it happes? Everything incapsulated in separate threads and works fast except this method.
Presumably you are using the browser http stack, in which case your browser will almost certainly have a limit on the number of connections you can make to a given subdomain at once- in many browser this is two.
A common trick, is to use multiple sub domains to serve content.
Here is some background about my app:
I am developing an Android app that will display a random quote or verse to the user. For this I am using an SQLite database. The size of the DB would be approximately 5K to 10K records, possibly increasing to upto 1M in later versions as new quotes and verses are added. Thus the user would need to update the DB as and when newer versions are of the app or DB are released.
After reading through some forums online, there seem to be two feasible ways I could provide the DB:
1. Bundle it along with the .APK file of the app, or
2. Upload it to my app's website from where users will have to download it
I want to know which method would be better (if there is yet another approach other than these, please do let me know).
After pondering this problem for some time, I have these thoughts regarding the above approaches:
Approach 1:
Users will obtain the DB along with the app, and won't have to download it separately. Installation would thereby be easier. But, users will have to reinstall the app every time there is a new version of the DB. Also, if the DB is large, it will make the installable too cumbersome.
Approach 2:
Users will have to download the full DB from the website (although I can provide a small, sample version of the DB via Approach 1). But, the installer will be simpler and smaller in size. Also, I would be able to provide future versions of the DB easily for those who might not want newer versions of the app.
Could you please tell me from a technical and an administrative standpoint which approach would be the better one and why?
If there is a third or fourth approach better than either of these, please let me know.
Thank you!
Andruid
I built a similar app for Android which gets periodic updates with data from a government agency. It's fairly easy to build an Android compatible db off the device using perl or similar and download it to the phone from a website; and this works rather well, plus the user gets current data whenever they download the app. It's also supposed to be possible to throw the data onto the sdcard if you want to avoid using primary data storage space, which is a bigger concern for my app which has a ~6Mb database.
In order to make Android happy with the DB, I believe you have to do the following (I build my DB using perl).
$st = $db->prepare( "CREATE TABLE \"android_metadata\" (\"locale\" TEXT DEFAULT 'en_US')");
$st->execute();
$st = $db->prepare( "INSERT INTO \"android_metadata\" VALUES ('en_US')");
$st->execute();
I have an update activity which checks weather updates are available and if so presents an "update now" screen. The download process looks like this and lives in a DatabaseHelperClass.
public void downloadUpdate(final Handler handler, final UpdateActivity updateActivity) {
URL url;
try {
close();
File f = new File(getDatabasePath());
if (f.exists()) {
f.delete();
}
getReadableDatabase();
close();
url = new URL("http://yourserver.com/" + currentDbVersion + ".sqlite");
URLConnection urlconn = url.openConnection();
final int contentLength = urlconn.getContentLength();
Log.i(TAG, String.format("Download size %d", contentLength));
handler.post(new Runnable() {
public void run() {
updateActivity.setProgressMax(contentLength);
}
});
InputStream is = urlconn.getInputStream();
// Open the empty db as the output stream
OutputStream os = new FileOutputStream(f);
// transfer bytes from the inputfile to the outputfile
byte[] buffer = new byte[1024 * 1000];
int written = 0;
int length = 0;
while (written < contentLength) {
length = is.read(buffer);
os.write(buffer, 0, length);
written += length;
final int currentprogress = written;
handler.post(new Runnable() {
public void run() {
Log.i(TAG, String.format("progress %d", currentprogress));
updateActivity.setCurrentProgress(currentprogress);
}
});
}
// Close the streams
os.flush();
os.close();
is.close();
Log.i(TAG, "Download complete");
openDatabase();
} catch (Exception e) {
Log.e(TAG, "bad things", e);
}
handler.post(new Runnable() {
public void run() {
updateActivity.refreshState(true);
}
});
}
Also note that I keep a version number in the filename of the db files, and a pointer to the current one in a text file on the server.
It sounds like your app and your db are tightly bound -- that is, the db is useless without the database and the database is useless without the app, so I'd say go ahead and put them both in the same .apk.
That being said, if you expect the db to change very slowly over time, but the app to change quicker, and you don't want your users to have to download the db with each new app revision, then you might want to unbundle them. To make this work, you can do one of two things:
Install them as separate applications, but make sure they share the same userID using the sharedUserId tag in the AndroidManifest.xml file.
Install them as separate applications, and create a ContentProvider for the database. This way other apps could make use of your database as well (if that is useful).
If you are going to store the db on your website then I would recommend that you just make rpc calls to your webserver and get data that way, so the device will never have to deal with a local database. Using a cache manager to avoid multiple lookups will help as well so pages will not have to lookup data each time a page reloads. Also if you need to update the data you do not have to send out a new app every time. Using HttpClient is pretty straight forward, if you need any examples please let me know
Can I do nested transactions in NHibernate, and how do I implement them? I'm using SQL Server 2008, so support is definitely in the DBMS.
I find that if I try something like this:
using (var outerTX = UnitOfWork.Current.BeginTransaction())
{
using (var nestedTX = UnitOfWork.Current.BeginTransaction())
{
... do stuff
nestedTX.Commit();
}
outerTX.Commit();
}
then by the time it comes to outerTX.Commit() the transaction has become inactive, and results in a ObjectDisposedException on the session AdoTransaction.
Are we therefore supposed to create nested NHibernate sessions instead? Or is there some other class we should use to wrap around the transactions (I've heard of TransactionScope, but I'm not sure what that is)?
I'm now using Ayende's UnitOfWork implementation (thanks Sneal).
Forgive any naivety in this question, I'm still new to NHibernate.
Thanks!
EDIT: I've discovered that you can use TransactionScope, such as:
using (var transactionScope = new TransactionScope())
{
using (var tx = UnitOfWork.Current.BeginTransaction())
{
... do stuff
tx.Commit();
}
using (var tx = UnitOfWork.Current.BeginTransaction())
{
... do stuff
tx.Commit();
}
transactionScope.Commit();
}
However I'm not all that excited about this, as it locks us in to using SQL Server, and also I've found that if the database is remote then you have to worry about having MSDTC enabled... one more component to go wrong. Nested transactions are so useful and easy to do in SQL that I kind of assumed NHibernate would have some way of emulating the same...
NHibernate sessions don't support nested transactions.
The following test is always true in version 2.1.2:
var session = sessionFactory.Open();
var tx1 = session.BeginTransaction();
var tx2 = session.BeginTransaction();
Assert.AreEqual(tx1, tx2);
You need to wrap it in a TransactionScope to support nested transactions.
MSDTC must be enabled or you will get error:
{"Network access for Distributed Transaction Manager (MSDTC) has been disabled. Please enable DTC for network access in the security configuration for MSDTC using the Component Services Administrative tool."}
As Satish suggested, nested transactions are not supported in NHibernate. I've not come across scenarios where nested transactions were needed, but certainly I've faced problems where I had to ignore creating transactions if other ones were already active in other units of work.
The blog link below provides an example implementation for NHibernate, but should also work for SQL server:
http://rajputyh.blogspot.com/2011/02/nested-transaction-handling-with.html
I've been struggling with this for a while now. Am going to have another crack at it.
I want to implement transactions in individual service containers - because that makes them self-contained - but then be able to nest a bunch of those service methods within a larger transaction and rollback the whole lot if necessary.
Because I'm using Rhino Commons I'm now going to try refactoring using the With.Transaction method. Basically it allows us to write code as if transactions were nested, though in reality there is only one.
For example:
private Project CreateProject(string name)
{
var project = new Project(name);
With.Transaction(delegate
{
UnitOfWork.CurrentSession.Save(project);
});
return project;
}
private Sample CreateSample(Project project, string code)
{
var sample = new Sample(project, code);
With.Transaction(delegate
{
UnitOfWork.CurrentSession.Save(sample);
});
return sample;
}
private void Test_NoNestedTransaction()
{
var project = CreateProject("Project 1");
}
private void TestNestedTransaction()
{
using (var tx = UnitOfWork.Current.BeginTransaction())
{
try
{
var project = CreateProject("Project 6");
var sample = CreateSample(project, "SAMPLE006", true);
}
catch
{
tx.Rollback();
throw;
}
tx.Commit();
}
}
In Test_NoNestedTransaction(), we are creating a project alone, without the context of a larger transaction. In this case, in CreateSample a new transaction will be created and committed, or rolled back if an exception occurs.
In Test_NestedTransaction(), we are creating both a sample and a project. If anything goes wrong, we want both to be rolled back. In reality, the code in CreateSample and CreateProject will run just as if there were no transactions at all; it is entirely the outer transaction that decides whether to rollback or commit, and does so based on whether an exception is thrown. Really that's why I'm using a manually created transaction for the outer transaction; so we I have control over whether to commit or rollback, rather than just defaulting to on-exception-rollback-else-commit.
You could achieve the same thing without Rhino.Commons by putting a whole lot of this sort of thing through your code:
if (!UnitOfWork.Current.IsInActiveTransaction)
{
tx = UnitOfWork.Current.BeginTransaction();
}
_auditRepository.SaveNew(auditEvent);
if (tx != null)
{
tx.Commit();
}
... and so on. But With.Transaction, despite the clunkiness of needing to create anonymous delegates, does that quite conveniently.
An advantage of this approach over using TransactionScopes (apart from the reliance on MSDTC) is that there ought to be just a single flush to the database in the final outer-transaction commit, regardless of how many methods have been called in-between. In other words, we don't need to write uncommitted data to the database as we go, we're always just writing it to the local NHibernate cache.
In short, this solution doesn't offer ultimate control over your transactions, because it doesn't ever use more than one transaction. I guess I can accept that, since nested transactions are by no means universally supported in every DBMS anyway. But now perhaps I can at least write code without worrying about whether we're already in a transaction or not.
That implementation doesn't support nesting, if you want nesting use Ayende's UnitOfWork implementation. Another problem with the implementation your are using (at least for web apps) is that it holds onto the ISession instance in a static variable.
I just rewrote our UnitOfWork yesterday for these reasons, it was originally based off of Gabriel's.
We don't use UnitOfWork.Current.BeginTransaction(), we use UnitofWork.TransactionalFlush(), which creates a separate transaction at the very end to flush all the changes at once.
using (var uow = UnitOfWork.Start())
{
var entity = repository.Get(1);
entity.Name = "Sneal";
uow.TransactionalFlush();
}