RODBC not freeing up memory after rm() and gc() - sql-server

I am connecting to a SQL server database in R using the RODBC package and making multiple calls to the database, doing something with the result, writing the result to disk and removing all variables. I then call gc() and close the channel before starting the next iteration. This is roughly shown below.
for(i in 1:1000){
ch <- odbcConnect("some dsn")
result = sqlQuery(ch, 'SELECT * from table where some_condition = i')
#some processing here...
saveRDS(result, 'result.rdata')
# this should remove all object in memory
rm(result)
gc()
odbcClose(ch)
}
My problem is that as the loop progresses, the amount of memory being used by R gets larger and larger (as revelaed in the windows Resouce Monitor), despite the total size of the objects in the global environment not increasing. Eventually, I use up all of the memory available on my machine (16gb, Windows 7, 64 bit) and the process crashes.
Why is this happenning?
How can I force R to free up the memory without restarting R?
Things I have tried that did not work:
multiple calls to gc() as suggested here https://stackoverflow.com/a/1467334/366502.
changing the buffsize, rows_at_time and believeNRows arguments in sqlQuery()
sessionInfo()
R version 3.2.0 (2015-04-16)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1
locale:
[1] LC_COLLATE=English_United Kingdom.1252 LC_CTYPE=English_United Kingdom.1252 LC_MONETARY=English_United Kingdom.1252 LC_NUMERIC=C LC_TIME=English_United Kingdom.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] data.table_1.9.4 DBI_0.3.1 rsqlserver_1.0 rClr_0.7-4 text_0.0.0.9000 stringr_1.0.0 plyr_1.8.3 RODBC_1.3-11 rvest_0.2.0 tomr_0.0.0.9000
loaded via a namespace (and not attached):
[1] Rcpp_0.12.0 lattice_0.20-31 XML_3.98-1.2 slam_0.1-32 grid_3.2.0 chron_2.3-47 R6_2.1.1 magrittr_1.5 httr_1.0.0 stringi_0.5-5 reshape2_1.4.1 NLP_0.1-8 Matrix_1.2-1 tools_3.2.0
[15] parallel_3.2.0 tm_0.6-2

Related

Julia - getting the version number of my module

I am writing a package for Julia and within the package I would like to be able to print the my package current version number. I can of course do that manually but I was looking for a way to read it directly from the module. the version number is written in the Project.toml file, but I have not been able to find a way to read it from the code. Any ideas?
You could use something like
julia> using Pkg
julia> loadedversion(m::Module) = VersionNumber(Pkg.TOML.parsefile(joinpath(string(first(methods(m.eval)).file), "..", "..", "Project.toml"))["version"])
loadedversion (generic function with 1 method)
julia> loadedversion(Atom)
v"0.12.8"
Some error handling would be prudent, of course, but that will give you the version of a package that's actually loaded (instead of the one specified in the current environment).
Possibly -
using Pkg
Pkg.TOML.parse(read("Project.toml", String))["version"]
-> "0.0.1"
using Pkg
function pkginfo(pkgname::AbstractString, key = "version")
# Check if Package Project
proj = Pkg.TOML.parsefile(Base.current_project())
proj["name"] == pkgname && return proj[key]
# If not in Project, pkgname must be in deps. Look in Manifest.toml
# An error will be thrown if Manifest.toml is missing
man = Pkg.TOML.parsefile(joinpath(Base.current_project() |> dirname, "Manifest.toml"));
for (pkg, info) in man
pkginfo = info |> first # Play with 'man' to see why
pkg == pkgname && return pkginfo[key]
end
end
This will give you the version (as string) of a package (or other data) in the current enviroment. But a package could be in a deeper enviroment in the stack, so...
Could it be that this is an answer?
using Pkg
Pkg.project().version
v"0.1.2"

xmlrpc.client.Fault when calling TestRun.update()

Kiwi version 6.0, tcms-api 5.0.
Given that 82 is a valid test run_id and 7 is a valid build_id for the test run's product in the Kiwi instance, run this Python snippet:
from tcms_api import TCMS
kiwi = TCMS()
kiwi.exec.TestRun.update(82, {'build' : 7})
Expect:
The test run's product build is updated from 1 (unspecified) to 7.
Result:
Exception has occurred: xmlrpc.client.Fault
<Fault -32603: "Internal error: 'status'">
There's no other call stack information, so I'm at a loss to further debug. I've tried updating a couple of different fields (manager and status) with the same result. I also get the same result if the value I'm trying to update is unknown/invalid.
Additional information: the equivalent call to TestCaseRun.update() API works. I.e., I can update the build information on a TestCaseRun instance.
#s-manke. This is a genuine bug. I've implemented a hot-fix here: https://github.com/kiwitcms/Kiwi/pull/553
so you can at least continue to use the API.
I am in the process of cutting a new version anyway so this hot-fix will go in. However the API will not process status or stop_date fields at the moment.

Using JanusGraph with Solr

Setting up JanusGraph i noticed the following in the console:
09:04:12,175 INFO ReflectiveConfigOptionLoader:173 - Loaded and initialized config classes: 10 OK out of 12 attempts in PT0.023S
09:04:12,230 INFO Reflections:224 - Reflections took 28 ms to scan 1 urls, producing 2 keys and 2 values
09:04:12,291 WARN GraphDatabaseConfiguration:1445 - Local setting index.search.index-name=entity (Type: GLOBAL_OFFLINE) is overridden by globally managed value (janusgraph). Use the ManagementSystem interface instead of the local configuration to control this setting.
09:04:12,294 WARN GraphDatabaseConfiguration:1445 - Local setting index.search.backend=solr (Type: GLOBAL_OFFLINE) is overridden by globally managed value (elasticsearch). Use the ManagementSystem interface instead of the local configuration to control this setting.
09:04:12,300 INFO CassandraThriftStoreManager:628 - Closed Thrift connection pooler.
and then i see the following:
Exception in thread "main" java.lang.IllegalArgumentException: Could not instantiate implementation: org.janusgraph.diskstorage.es.ElasticSearchIndex
How do i stop using elasticsearch and switch to Solr?
My properties file is as follows:
index.search.backend=solr
index.search.directory=/path/to/directory/for/solr/index/something
index.search.index-name=something
index.search.solr.mode=http
index.search.solr.http-urls=http://127.0.0.1:8983/solr
storage.backend=cassandrathrift
storage.hostname=127.0.0.1
cache.db-cache = true
cache.db-cache-clean-wait = 20
cache.db-cache-time = 180000
cache.db-cache-size = 0.25
The answer to this basically the same as this one for Titan. JanusGraph was forked from Titan.
You are probably trying to connect to an existing graph that was previously configured to use Elasticsearch. By default, the keyspace is named janusgraph.
1) You could connect to a different keyspace by updating conf/janusgraph-cassandra.properties
gremlin.graph=org.janusgraph.core.JanusGraphFactory
storage.backend=cassandrathrift
storage.hostname=127.0.0.1
storage.cassandra.keyspace=mygraph
2) You could drop the existing keyspace. If you used bin/janusgraph.sh start from the quick start directions (which starts a single node Cassandra and a single node Elasticsearch),
bin/janusgraph.sh clean
Or if you have a standalone Cassandra installation:
$CASSANDRA_HOME/bin/cqlsh -e 'drop keyspace if exists janusgraph'
Then you would be able to connect with the default conf/janusgraph-cassandra.properties.

Failing lein uberjar when reading database from configuration

I am writing a Ring / Compojure app with Clojure that fetches content for pages from database. To be able to create tests for how the content is displayed, I created prod and dev environments and when using dev environment, a mock database is used instead of the production database. I achieve this by reading the database from another file and giving it as a parameter to my routes. Here's a simplified version:
(defn www-routes [db]
(defroutes www-routes
(GET "/" [] ...)))
(def config (delay (load-file (.getFile (resource "config.clj")))))
(defn db []
(if (= "dev" (:database #(force config)))
'kipsu.db-mock
'kipsu.database))
(def app (routes (www-routes (db)))
The setup is largely taken from the example here, with the addition of setting the database as a parameter.
This setup works great with running the tests with the mock database and displaying real content on prod environment. Things run fine when I start a lein server locally, run tests or any of the functions in lein repl. My problem comes when I'd like to create an uberjar for deploying the changes on my server.
This is where I get a NullPointerException when compiling, starting from (db) function call inside the def app. I've tried debugging with poor success and am not even 100% sure where the actual error is. All I know is the db function is never even called. Here's the stack trace:
Compiling kipsu.jdbc.json
Compiling kipsu.database
Compiling kipsu.api_converter
Compiling kipsu.web
java.lang.NullPointerException, compiling:(web.clj:124:29)
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3628)
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3622)
at clojure.lang.Compiler$BodyExpr.eval(Compiler.java:5879)
at clojure.lang.Compiler$DefExpr.eval(Compiler.java:439)
at clojure.lang.Compiler.compile1(Compiler.java:7323)
at clojure.lang.Compiler.compile(Compiler.java:7390)
at clojure.lang.RT.compile(RT.java:399)
at clojure.lang.RT.load(RT.java:444)
at clojure.lang.RT.load(RT.java:412)
at clojure.core$load$fn__5448.invoke(core.clj:5866)
at clojure.core$load.doInvoke(core.clj:5865)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5671)
at clojure.core$compile$fn__5453.invoke(core.clj:5877)
at clojure.core$compile.invoke(core.clj:5876)
at user$eval9$fn__16.invoke(form-init1768231915654429312.clj:1)
at user$eval9.invoke(form-init1768231915654429312.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6782)
at clojure.lang.Compiler.eval(Compiler.java:6772)
at clojure.lang.Compiler.load(Compiler.java:7227)
at clojure.lang.Compiler.loadFile(Compiler.java:7165)
at clojure.main$load_script.invoke(main.clj:275)
at clojure.main$init_opt.invoke(main.clj:280)
at clojure.main$initialize.invoke(main.clj:308)
at clojure.main$null_opt.invoke(main.clj:343)
at clojure.main$main.doInvoke(main.clj:421)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at clojure.lang.Var.invoke(Var.java:383)
at clojure.lang.AFn.applyToHelper(AFn.java:156)
at clojure.lang.Var.applyTo(Var.java:700)
at clojure.main.main(main.java:37)
Caused by: java.lang.NullPointerException
at kipsu.web$fn__4568.invoke(web.clj:115)
at clojure.lang.Delay.deref(Delay.java:37)
at clojure.lang.Delay.force(Delay.java:27)
at clojure.core$force.invoke(core.clj:730)
at kipsu.web$db.invoke(web.clj:118)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3623)
... 30 more
Exception in thread "main" java.lang.NullPointerException, compiling (web.clj:124:29)
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3628)
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3622)
at clojure.lang.Compiler$BodyExpr.eval(Compiler.java:5879)
at clojure.lang.Compiler$DefExpr.eval(Compiler.java:439)
at clojure.lang.Compiler.compile1(Compiler.java:7323)
at clojure.lang.Compiler.compile(Compiler.java:7390)
at clojure.lang.RT.compile(RT.java:399)
at clojure.lang.RT.load(RT.java:444)
at clojure.lang.RT.load(RT.java:412)
at clojure.core$load$fn__5448.invoke(core.clj:5866)
at clojure.core$load.doInvoke(core.clj:5865)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at clojure.core$load_one.invoke(core.clj:5671)
at clojure.core$compile$fn__5453.invoke(core.clj:5877)
at clojure.core$compile.invoke(core.clj:5876)
at user$eval9$fn__16.invoke(form-init1768231915654429312.clj:1)
at user$eval9.invoke(form-init1768231915654429312.clj:1)
at clojure.lang.Compiler.eval(Compiler.java:6782)
at clojure.lang.Compiler.eval(Compiler.java:6772)
at clojure.lang.Compiler.load(Compiler.java:7227)
at clojure.lang.Compiler.loadFile(Compiler.java:7165)
at clojure.main$load_script.invoke(main.clj:275)
at clojure.main$init_opt.invoke(main.clj:280)
at clojure.main$initialize.invoke(main.clj:308)
at clojure.main$null_opt.invoke(main.clj:343)
at clojure.main$main.doInvoke(main.clj:421)
at clojure.lang.RestFn.invoke(RestFn.java:421)
at clojure.lang.Var.invoke(Var.java:383)
at clojure.lang.AFn.applyToHelper(AFn.java:156)
at clojure.lang.Var.applyTo(Var.java:700)
at clojure.main.main(main.java:37)
Caused by: java.lang.NullPointerException
at kipsu.web$fn__4568.invoke(web.clj:115)
at clojure.lang.Delay.deref(Delay.java:37)
at clojure.lang.Delay.force(Delay.java:27)
at clojure.core$force.invoke(core.clj:730)
at kipsu.web$db.invoke(web.clj:118)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3623)
... 30 more
Compilation failed: Subprocess failed
I'm not the most fluent with Clojure and am working with this app to learn more. Any help for steering me at the right direction from here is greatly appreciated!
I think your problem is because def will execute at compile time. You've done the right thing in (def config) and (defn db), but (def app) is going to cause compile time errors if it cannot find your file. To understand why let's look at def.
(def hello (println "hello"))
If you try to compile this code you'll see "hello" printed to your console at compile time and at runtime the var hello will have the value nil.
(def hello (delay (println "hello"))
(def world #hello)
The var hello now won't get evaluated at compile time, but by introducing the var world we get the exact same problem.
Now back to your specific problem. You don't want your configuration to get read at compile time, and you don't want your configuration to have to read a file from disk every single time you need it.
Not reading your configuration at compile time makes me think that maybe it should be a function. If it is then a function you can simply use memoize to ensure it doesn't read from disk every time you call that function.

memory leak detected when running on linux:

I tried to increase the no.of listeners using Event Emitter. But it's not working. The same is running with no warnings in windows.
(node) warning: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit.
Trace
at EventEmitter.addListener (events.js:160:15)
at Server.connect (/dir/node_modules/mongoose/node_modules/mongodb/lib/server.js:291:17)
at Db.open (/dir/node_modules/mongoose/node_modules/mongodb/lib/db.js:190:19)
at MongoStore._open_database (/dir/node_modules/connect-mongo/lib/connect-mongo.js:182:15)
at MongoStore._get_collection (/dir/node_modules/connect-mongo/lib/connect-mongo.js:177:14)
at /dir/node_modules/connect-mongo/lib/connect-mongo.js:194:16
at /dir/node_modules/mongoose/node_modules/mongodb/lib/db.js:200:5
at connectHandler (/dir/node_modules/mongoose/node_modules/mongodb/lib/server.js:272:7)
at g (events.js:180:16)
at EventEmitter.emit (events.js:95:17)
Solved...
Modifying max no.of listeners did not work.
The problem is with new versions of mongoose/mongodb.
When i saw mongodb server, i noticed that connections are created to the DB in a continuous loop (may be due to some problem with new ones).
I switched them back to the previous versions in package.json, cleared cache and installed dependencies again. Now its working.

Resources