Automatic failover - Mongoid 4.0.2 - mongoid

Trying to test automatic failover using Mongoid 4.0.2 gem and using MongoDB 2.4.3
To simulate this I'm using this test code:
require 'mongoid'
class TestClass
include Mongoid::Document
store_in collection: "test", database: "test"
field :uuid, type: String
end
Mongoid.load!("config/mongoid.yml", :test)
batch = (1..100).map { |x| TestClass.new({ uuid: x }) }
batch.each_with_index { |x, i|
begin
x.save
sleep(5.seconds)
puts "Saved #{i} records" if i%10 == 0
rescue Exception => e
puts e.message
end
}
In between saves, I jumped on my MongoDB and did rs.stepDown() on the primary node of my Mongo cluster, unfortunately this results in the following errors in my test app:
See https://github.com/mongodb/mongo/blob/master/docs/errors.md
for details about this error.
Moped::Errors::OperationFailure
The operation: #<Moped::Protocol::Command
#length=68
#request_id=192
#response_to=0
#op_code=2004
#flags=[]
#full_collection_name="test.$cmd"
#skip=0
#limit=-1
#selector={:getlasterror=>1, :w=>1}
#fields=nil>
failed with error 10058: "not master"
My Mongoid configuration looks like thus:
test:
sessions:
default:
database: test_db
hosts:
- 192.168.1.10:27017
- 192.168.1.11:27017
options:
max_retries: 10
retry_interval: 1
Any idea what I'm doing wrong here? I thought the Mongoid driver would automatically detect changes in the cluster and automatically retry the request after it updates the cluster state on the client / Ruby side?

Related

In Python, how do I construct a URL to test if my SQL Server instance is running?

I'm using Python 3.8 with the pytest-docker-compose plugin -- https://pypi.org/project/pytest-docker-compose/ . Does anyone know how to write a URL that would eventually tell me if my SQL Server is running?
I have this docker-compose.yml file
version: "3.2"
services:
sql-server-db:
build: ./
container_name: sql-server-db
image: microsoft/mssql-server-linux:2017-latest
ports:
- "1433:1433"
environment:
SA_PASSWORD: "password"
ACCEPT_EULA: "Y"
but I don't know what URL to pass to my Retry object to test that the server is running. This fails ...
import pytest
import requests
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter
...
#pytest.fixture(scope="function")
def wait_for_api(function_scoped_container_getter):
"""Wait for sql server to become responsive"""
request_session = requests.Session()
retries = Retry(total=5,
backoff_factor=0.1,
status_forcelist=[500, 502, 503, 504])
request_session.mount('http://', HTTPAdapter(max_retries=retries))
service = function_scoped_container_getter.get("sql-server-db").network_info[0]
api_url = "http://%s:%s/" % (service.hostname, service.host_port)
assert request_session.get(api_url)
return request_session, api_url
with this exception
raise ConnectionError(e, request=request)
E requests.exceptions.ConnectionError: HTTPConnectionPool(host='0.0.0.0', port=1433): Max retries exceeded with url: / (Caused by ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')))
if connection.is_connected():
db_Info = connection.get_server_info()
print("Connected to MySQL Server version ", db_Info)
cursor = connection.cursor()
cursor.execute("select database();")
record = cursor.fetchone()
print("You're connected to database: ", record)
you could use something like this and it would output if it was connected
Here is a sample function that will retry to connect to the DB and won't return until it has successfully connected or the defined maxRetries is reached:
def waitDb(server, database, username, password, maxAttempts, waitBetweenAttemptsSeconds):
"""
Returns True if the connection is successfully established before the maxAttempts number is reached
Conversely returns False
pyodbc.connect has a built-in timeout. Use a waitBetweenAttemptsSeconds greater than zero to add a delay on top of this timeout
"""
for attemptNumber in range(maxAttempts):
cnxn = None
try:
cnxn = pyodbc.connect('DRIVER={ODBC Driver 17 for SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)
cursor = cnxn.cursor()
except Exception as e:
print(traceback.format_exc())
finally:
if cnxn:
print("The DB is up and running: ")
return True
else:
print("DB not running yet on attempt numer " + str(attemptNumber))
time.sleep(waitBetweenAttemptsSeconds)
print("Max attempts waiting for DB to come online exceeded")
return False
I wrote a minimal example here: https://github.com/claudiumocanu/flow-pytest-docker-compose-wait-mssql.
I included the three actions that can be executed independently, but you can jump to the last step specifically for what you asked:
1. Connected from python to the mssql launched by the compose-file:
For me it was quite annoying to find and install the appropriate ODBC Driver and its dependencies - ODBC Driver 17 for SQL Server worked the best for me on Ubuntu 18.
To perform only this step, docker-compose up the the docker-compose.yml in my example, then run the example-connect.py
2. Created a function that attempts to connect to the DB with a maxAttemptsNumber and a delay between retries:
Just run this example-waitDb.py. You can play with the maxAttempts and the delayBetweenAttempts values, then bring up the database at randomly, to test it.
3. Put everything together in the test_db.py test suite:
the waitDb function described above.
same wrapper and annotations that you provided in your example to spin-up the resources defined in the compose-file
a dummy integration test that will not be executed before waitDb returns (if you want to block this tests completely, you can throw instead of returning False from the waitDb function)
PS: Keep using ENVs/vault etc rather than storing the real passwords like I did for the dummy example.

AWS: Why does my RDS instance keep starting after I turned it off?

I have an RDS database instance on AWS and have turned it off for now. However, every few days it starts up on its own. I don't have any other services running right now.
There is this event in my RDS log:
"DB instance is being started due to it exceeding the maximum allowed time being stopped."
Why is there a limit to how long my RDS instance can be stopped? I just want to put my project on hold for a few weeks, but AWS won't let me turn off my DB? It costs $12.50/mo to have it sit idle, so I don't want to pay for this, and I certainly don't want AWS starting an instance for me that does not get used.
Please help!
That's a limitation of this new feature.
You can stop an instance for up to 7 days at a time. After 7 days, it will be automatically started. For more details on stopping and starting a database instance, please refer to Stopping and Starting a DB Instance in the Amazon RDS User Guide.
You can setup a cron job to stop the instance again after 7 days. You can also change to a smaller instance size to save money.
Another option is the upcoming Aurora Serverless which stops and starts for you automatically. It might be more expensive than a dedicated instance when running 24/7.
Finally, there is always Heroku which gives you a free database instance that starts and stops itself with some limitations.
You can also try saving the following CloudFormation template as KeepDbStopped.yml and then deploy with this command:
aws cloudformation deploy --template-file KeepDbStopped.yml --stack-name stop-db --capabilities CAPABILITY_IAM --parameter-overrides DB=arn:aws:rds:us-east-1:XXX:db:XXX
Make sure to change arn:aws:rds:us-east-1:XXX:db:XXX to your RDS ARN.
Description: Automatically stop RDS instance every time it turns on due to exceeding the maximum allowed time being stopped
Parameters:
DB:
Description: ARN of database that needs to be stopped
Type: String
AllowedPattern: arn:aws:rds:[a-z0-9\-]+:[0-9]+:db:[^:]*
Resources:
DatabaseStopperFunction:
Type: AWS::Lambda::Function
Properties:
Role: !GetAtt DatabaseStopperRole.Arn
Runtime: python3.6
Handler: index.handler
Timeout: 20
Code:
ZipFile:
Fn::Sub: |
import boto3
import time
def handler(event, context):
print("got", event)
db = event["detail"]["SourceArn"]
id = event["detail"]["SourceIdentifier"]
message = event["detail"]["Message"]
region = event["region"]
rds = boto3.client("rds", region_name=region)
if message == "DB instance is being started due to it exceeding the maximum allowed time being stopped.":
print("database turned on automatically, setting last seen tag...")
last_seen = int(time.time())
rds.add_tags_to_resource(ResourceName=db, Tags=[{"Key": "DbStopperLastSeen", "Value": str(last_seen)}])
elif message == "DB instance started":
print("database started (and sort of available?)")
last_seen = 0
for t in rds.list_tags_for_resource(ResourceName=db)["TagList"]:
if t["Key"] == "DbStopperLastSeen":
last_seen = int(t["Value"])
if time.time() < last_seen + (60 * 20):
print("database was automatically started in the last 20 minutes, turning off...")
time.sleep(10) # even waiting for the "started" event is not enough, so add some wait
rds.stop_db_instance(DBInstanceIdentifier=id)
print("success! removing auto-start tag...")
rds.add_tags_to_resource(ResourceName=db, Tags=[{"Key": "DbStopperLastSeen", "Value": "0"}])
else:
print("ignoring manual database start")
else:
print("error: unknown database event!")
DatabaseStopperRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Action:
- sts:AssumeRole
Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Policies:
- PolicyName: Notify
PolicyDocument:
Version: '2012-10-17'
Statement:
- Action:
- rds:StopDBInstance
Effect: Allow
Resource: !Ref DB
- Action:
- rds:AddTagsToResource
- rds:ListTagsForResource
- rds:RemoveTagsFromResource
Effect: Allow
Resource: !Ref DB
Condition:
ForAllValues:StringEquals:
aws:TagKeys:
- DbStopperLastSeen
DatabaseStopperPermission:
Type: AWS::Lambda::Permission
Properties:
Action: lambda:InvokeFunction
FunctionName: !GetAtt DatabaseStopperFunction.Arn
Principal: events.amazonaws.com
SourceArn: !GetAtt DatabaseStopperRule.Arn
DatabaseStopperRule:
Type: AWS::Events::Rule
Properties:
EventPattern:
source:
- aws.rds
detail-type:
- "RDS DB Instance Event"
resources:
- !Ref DB
detail:
Message:
- "DB instance is being started due to it exceeding the maximum allowed time being stopped."
- "DB instance started"
Targets:
- Arn: !GetAtt DatabaseStopperFunction.Arn
Id: DatabaseStopperLambda
It has worked for at least one person. If you have issues please report here.

sh.isBalancerRunning(): false

I am trying to shard a mongodb database like this:
1- Start each member of the shard replica set
mongod --shardsvr --port 27100 --replSet r1 --dbpath <some_path>\shardsvr\shardsvr1
mongod --shardsvr --port 27200 --replSet r2 --dbpath <some_path>\shardsvr\shardsvr2
2- Start each member of the config server replica set
mongod --configsvr --port 27020 --replSet cfg1 --dbpath <some_path>\configsvr\configsvr1
3- Connect to config server replica set
mongo --port 27020
4- Initiate the replica set
conf = {
_id: "cfg1",
members: [
{
_id:0,
host: "localhost:27020"
}
]
}
rs.initiate(conf)
5- Start the mongos and specify the --configdb parameter
mongos --configdb cfg1/localhost:27020 --port 28000
6- Initiate the replica set of each shard
mongo --port 27100
var config = {_id: "r1", members: [{_id:0, host:"localhost:27100"}]}
rs.initiate(config)
exit
mongo --port 27200
var config = {_id: "r2", members: [{_id:0, host:"localhost:27200"}]}
rs.initiate(config)
exit
7- Connect to mongos to add shards
mongo --port 28000
sh.addShard("r1/localhost:27100")
sh.addShard("r2/localhost:27200")
8- Add some data
use sharddb
for (i = 10000; i < 30000; i++){
db.example.insert({
author: "author" + i,
post_title: "Blog Post by Author " + i,
date: new Date()
});
}
db.example.count()
9- Enable sharding
sh.enableSharding("sharddb")
10- Create the index as part of sh.shardCollection()
db.example.ensureIndex({author : 1}, true)
sh.shardCollection("sharddb.example", {author: 1})
11- Check if balancer is running
sh.isBalancerRunning()
However, in this step, I get a false as response, and I dont know what I did wrong to get this. I followed the steps of this tutorial
With only 20000 documents that are ~100 bytes each, there is probably only 1 chunk.
Check with
use sharddb
db.printShardingStatus()
I repeated the steps you listed above, and got the following result:
{ "_id" : "sharddb", "primary" : "shard02", "partitioned" : true }
sharddb.example
shard key: { "author" : 1 }
unique: false
balancing: true
chunks:
shard02 1
{ "author" : { "$minKey" : 1 } } -->> { "author" : { "$maxKey" : 1 } } on : shard02 Timestamp(1, 0)
The mongos will monitor what it has added to each chunk, and notify the config server to consider splitting when it has seen enough data added. Then the balancer will automatically be activated when one shard contains several more chunks than another.
If you insert enough documents to trigger automatic splitting, or manually split the chunk, the balancer will begin doing its thing.

Sunspot.index(Model.all) doesn't re-index, though bundle exec rake sunspot:reindex does

I am using the following gems:
sunspot_mongo (1.2.1)
sunspot_solr (2.2.3)
I am not re-indexing with callbacks after creation, because I am bulk inserting objects with db.collection.insert().
Instead I want to re-index products a few times per day, e.g. in a recurring Sidekiq job.
According to the sunspot_solr README I can run:
Sunspot.index(Model.all)
According to the logs, this does set a re-indexing in motion:
SOLR Request (2.7ms) [ path=update parameters={} ]
SOLR Request (9.7ms) [ path=update parameters={} ]
...
but it does not seem to result in anything becoming searchable. Any idea why this is?
Fyi, I can run this rake task with a successfull result:
bundle exec rake sunspot:reindex
but I can't put that one into a Sidekiq job.
For the record, this is how my Model looks like:
class Model
include Mongoid::Document
include Sunspot::Mongo
searchable auto_index: false do
string :_id, :stored => true
text :name, :stored => true
end
end

Problem with grails web app running in production: "No such property: save for class: JsecRole"

I've got a grails 1.1 web app running great in development but when I try and run it in production with
an sqlserver database it crashes in a weird way.
The relevant part of my datasource.groovy is as follows:
environments {
development {
dataSource {
dbCreate = "create-drop" // one of 'create', 'create-drop','update'
url = "jdbc:hsqldb:mem:devDB"
}
}
test {
dataSource {
dbCreate = "update"
url = "jdbc:hsqldb:mem:testDb"
}
}
production {
dataSource {
dbCreate = "update"
driverClassName = "com.microsoft.sqlserver.jdbc.SQLServerDriver"
endUsername = "sa"
password = "pw4db"
url = "jdbc:sqlserver://localhost:1433;databaseName=ReleasePlanner;selectMethod=cursor"
The error message I receive is:
Message: No such property: save for class: JsecRole
Caused by: groovy.lang.MissingPropertyException: No such property: save for class: JsecRole
Class: ProjectController
At Line: [28]
Code Snippet:
27: println "###about to create project roles"
28: userManagerService.createProjectRoles(project)
29: userManagerService.addUserToProject(session.user.id.toString(), project, 'owner')
}
}
}
The stacktrace is as follows:
org.codehaus.groovy.runtime.InvokerInvocationException: groovy.lang.MissingPropertyException: No such property: save for class: JsecRole
at org.jsecurity.web.servlet.JSecurityFilter.doFilterInternal(JSecurityFilter.java:382)
at org.jsecurity.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:180)
Caused by: groovy.lang.MissingPropertyException: No such property: save for class: JsecRole
at UserManagerService.createProjectRoles(UserManagerService.groovy:9)
at UserManagerService$$FastClassByCGLIB$$6fa73713.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
at UserManagerService$$EnhancerByCGLIB$$fcf60984.createProjectRoles(<generated>)
at UserManagerService$createProjectRoles.call(Unknown Source)
at ProjectController$_closure4.doCall(ProjectController.groovy:28)
at ProjectController$_closure4.doCall(ProjectController.groovy)
... 2 more
Any help is appreciated.
Thanks
Sarah
I fixed my problem by deleting my database and creating a new database. I think some of the fields in my database weren't mapping correctly as I changed my domain objects. The error didn't really point me in this direction though!
Sarah
This problem is discussed in this thread on the Grails mailing list. It is supposed to be fixed in Grails 1.2. A workaround for earlier versions of Grails is to add the following to Bootstrap.groovy
JsecRole.get(-1)

Resources