How do I connect Ecto to CockroachDB Serverless? - database

I'd like to use CockroachDB Serverless for my Ecto application. How do I specify the connection string?
I get an error like this when trying to connect.
[error] GenServer #PID<0.295.0> terminating
** (Postgrex.Error) FATAL 08004 (sqlserver_rejected_establishment_of_sqlconnection) codeParamsRoutingFailed: missing cluster name in connection string
(db_connection 2.4.1) lib/db_connection/connection.ex:100: DBConnection.Connection.connect/2
CockroachDB Serverless says to connect by including the cluster name in the connection string, like this:
postgresql://username:<ENTER-PASSWORD>#free-tier.gcp-us-central1.cockroachlabs.cloud:26257/defaultdb?sslmode=verify-full&sslrootcert=$HOME/.postgresql/root.crt&options=--cluster%3Dcluster-name-1234
but I'm not sure how to get Ecto to create this connection string via its configuration.

The problem is that Postgrex is not able to parse all of the information from the connection URL - notable the SSL configuration. The solution is to specify the connection parameters explicitly, including the cacertfile SSL option. Assuming that you have downloaded your cluster's CA certificate to priv/certs/ca-cert.crt, you can use the following config as a template:
config :my_app, MyApp.Repo,
username: "my_user",
password: "my_password",
database: "defaultdb",
hostname: "free-tier.gcp-us-central1.cockroachlabs.cloud",
port: "26257",
ssl: true,
ssl_opts: [
cacertfile: Path.expand("priv/certs/ca-cert.crt"),
],
parameters: [options: "--cluster=my-cluster-123"]
Possible Other Issues
Table Locking
Since that CockroachDB also does not support the locking that Ecto/Postgrex attempts on the migration table, the :migration_lock config needs to be disabled as well:
config :my_app, MyApp.Repo,
# ...
migration_lock: false
Auth generator
Finally, the new phx.gen.auth generator defaults to using the citext extension for storing a user's email address in a case-insensitive manner. The line in the generated migration that executes CREATE EXTENSION IF NOT EXISTS citext should be removed, and the column type for the :email field should be changed from :citext to :string.

This configuration allows Ecto to connect to CockroachDB Serverless correctly:
config :myapp, MyApp.repo,
username: "username",
password: "xxxx",
database: "defaultdb",
hostname: "free-tier.gcp-us-central1.cockroachlabs.cloud",
port: 26257,
ssl: true,
ssl_opts: [
cert_pem: "foo.pem",
key_pem: "bar.pem"
],
show_sensitive_data_on_connection_error: true,
pool_size: 10,
parameters: [
options: "--cluster=cluster-name-1234"
]

Related

Receiving error when using mix ecto.create

This is my configuration for the database:
config :hello, Hello.Repo,
username: "postgres",
password: "admin",
hostname: "localhost",
database: "hello_dev_postgres",
show_sensitive_data_on_connection_error: true,
pool_size: 10
config :hello, Hello.MRepo,
adapter: Tds.Ecto,
username: "sa",
password: "server",
hostname: "localhost",
instance: "SERVER",
port: 1433,
database: "hello_dev_mssql",
show_sensitive_data_on_connection_error: true,
pool_size: 10
And this is my two repo:
defmodule Hello.MRepo do
use Ecto.Repo,
otp_app: :hello,
adapter: Ecto.Adapters.Tds
end
defmodule Hello.Repo do
use Ecto.Repo,
otp_app: :hello,
adapter: Ecto.Adapters.Postgres
end
When I run mix ecto.create, I get this error:
16:37:43.867 [error] GenServer #PID<0.309.0> terminating
** (Tds.Error) tcp connect: econnrefused
(db_connection 2.4.2) lib/db_connection/connection.ex:100: DBConnection.Connection.connect/2
(connection 1.1.0) lib/connection.ex:622: Connection.enter_connect/5
(stdlib 3.17) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Last message: nil
State: Tds.Protocol
(Mix) The database for Hello.MRepo couldn't be created: killed
Have you enabled TCP in your database configuration file?
There is a similar post on the elixirforum where someone had the same problem as you and enabling TCP in one of the configuration file helped them.

Elixir: Ecto/Tds error when connecting to SQL Server database

I've written an elixir application that connects to two different databases, and I have it working nicely in my remote database dev environment -- connecting with the same exact setup to a different database that is across my vpn. When I run it against a different remote SQL Server database, I get a strange error:
** (DBConnection.ConnectionError) bad return value: {:error, %Tds.Protocol{env: %{trans: <<0>>}, itcp: nil, opts: [password:
:REDACTED, idle_timeout: 5000, name: EventBridge.C3poRepo.Pool,
otp_app: :event_bridge, repo: EventBridge.C3poRepo, timeout: 15000,
pool_timeout: 5000, adapter: Tds.Ecto, username: :ALSO_REDACTED,
database: :REDACTED, hostname: :REDACTED, pool:
DBConnection.Poolboy], pak_data: "", pak_header: "", query:
%Tds.Query{handle: 1, statement: nil}, result: %Tds.Result{columns:
[], command: nil, num_rows: 0, rows: []}, sock: {:gen_tcp,
#Port<0.6472>}, state: :ready, tail: "", transaction: nil, usock: nil}}
The credentials I've got in my configuration for this database are correct--I verified them manually with DBeaver. I believe I'm capable of debugging this but I've run into a bit of that error message that I don't understand. Specifically:
{:error, %Tds.Protocol{env: %{trans: <<0>>}
For the sake of completeness, here the deps section in mix.exs:
defp deps do
[
{:tds_ecto, "~> 2.0.3"},
{:postgrex, ">= 0.0.0"},
{:gen_stage, "~> 0.12"},
{:timex, "~> 3.1"},
{:timex_ecto, "~> 3.0"},
{:poison, "~> 3.1"},
{:ecto, "~> 2.1"}
]
end
What I'm asking is for some insight into what that part of the error message means, or perhaps even some more general feedback on the entire error output.
Here are the two instances of configuration for the Repo in question. First the one that works:
config :event_bridge, EventBridge.FirstRepo,
adapter: Tds.Ecto,
database: "first_repo_development",
username: "testuser",
password: "password_redacted",
hostname: "server_one"
And here is the configuration for the one that doesn't work:
config :event_bridge, EventBridge.FirstRepo,
adapter: Tds.Ecto,
database: "first_repo_staging",
username: "staging_username",
password: "password_redacted",
hostname: "server_twelve"
Both work in DBeaver.
Add
pool_size:
in the db config and try again

SolrCtl instance using JAAS option

I am using Cloudera cluster and it is kerberosed with tsl and ssl enabled. I am trying to use create an instance using the --jaas option in solrctl command but it is not working.
The solrctl command I am using is below
solrctl --jaas jaas-client.conf instancedir --create testindex3 /home/myuserid/testindex3
The jass-client.conf file is below
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="<PATH TO KEYTAB>/username.keytab"
storeKey=true
useTicketCache=false
debug=true
principal="fully.qualified.domain.name#<YOUR-REALM";
}
For the above command I am getting the error
Uploading configs from /user/myuserid/testindex3/conf to ZookeeperHost1:2181,ZookeeperHost2:2181,ZookeeperHost3:2181/solr. This may take up to a minute.
Debug is true storeKey true useTicketCache false useKeyTab true doNotPrompt false ticketCache is null isInitiator true KeyTab is /username.keytab refreshKrb5Config is false principal is fully.qualified.domain.name#
Error: can't upload configuration
I am not sure why the instance is not created and what i am missing. Is there anything wrong in my jaas.conf file. Please advise
Note: I haven't used "user/fully.qualified.domain.name#

Spring Turbine dashboard not working

I am facing some issue while working on turbine dashboard. As I am able to get turbine stream for given cluster but not able to see anything on dashboard as it is just getting loaded as shown in below screenshots. Kindly help if any configuration is missing.
Below are my configurations:
config.properties
turbine.aggregator.clusterConfig=SpringHystrixDemo2
turbine.instanceUrlSuffix=:9080/hystrix.stream
turbine.EurekaInstanceDiscovery.hystrix2.instances=localhost
InstanceDiscovery.impl=com.netflix.turbine.discovery.EurekaInstanceDiscovery.class
turbine.InstanceMonitor.eventStream.skipLineLogic.enabled=false
Application.yml
server:
port: 8080
turbine:
aggregator:
clusterConfig: SPRINGHYSTRIXDEMO2
clusterNameExpression: new String("default")
appConfig: SpringHystrixDemo2
InstanceMonitor:
eventStream:
skipLineLogic:
enabled: false
bootstrap.yml
spring:
application:
name: SpringTurbine
cloud:
config:
discovery:
enabled: true
eureka:
instance:
nonSecurePort: ${server.port:8080}
client:
serviceUrl:
defaultZone: http://${eureka.host:localhost}:${eureka.port:8761}/eureka/
Application.java
#SpringBootApplication
#EnableHystrix
#EnableEurekaClient
#EnableHystrixDashboard
#EnableTurbine
public class DemoApplication {
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
}
For cluster SpringHystrixDemo2 I have configured it in different application running on other port:
application.yml -
server:
port: 9080
hystrix:
command:
RemoteMessageClientCommand:
execution:
isolation:
thread:
timeoutInMilliseconds: 5000
RemoteMessageAnnotationClient:
execution:
isolation:
thread:
timeoutInMilliseconds: 5000
bootstrap.yml
spring:
application:
name: SpringHystrixDemo2
cloud:
config:
enabled: true
discovery:
enabled: true
serviceId: SPRINGCONFIGSERVER
eureka:
instance:
nonSecurePort: ${server.port:9080}
client:
serviceUrl:
defaultZone: http://${eureka.host:localhost}:${eureka.port:8761}/eureka/
Application.java - this is from hystrix dashboard service.
#SpringBootApplication
#EnableHystrix
#EnableHystrixDashboard
#EnableEurekaClient
#EnableDiscoveryClient
public class DemoApplication {
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
}
I have configured eureka server on 8761 port. which is lisening all other eureka client.as
Eureka server
This is how I am not able to see any turbine dashboard. as it is just getting loaded.
turbine stream view
First thing that comes to my mind is that you could declare a management endpoint like this:
management:
port: 9081
contextPath: /management
Then the turbine stream would be accesible via {yourHost}:9081/management/turbine.stream, while hystrix dashboard will be served under {yourhost}:9080/hystrix
From what I have read and known, from your configuration
turbine.aggregator.clusterConfig=SpringHystrixDemo2
turbine.instanceUrlSuffix=:9080/hystrix.stream
turbine.EurekaInstanceDiscovery.hystrix2.instances=localhost InstanceDiscovery.impl=com.netflix.turbine.discovery.EurekaInstanceDiscovery.class
turbine.InstanceMonitor.eventStream.skipLineLogic.enabled=false
following maybe the problem(s).
You are short of a few configs and maybe you have some extra configs too.
You do not need "turbine.EurekaInstanceDiscovery.hystrix2.instances" unless you really have multiple instances.
You do not need "turbine.InstanceMonitor.eventStream.skipLineLogic.enabled" because its false by default, and its required if you want it to be true when u you have high latency.
you need "turbine.appConfig=". In your case i think its something like SpringHystrixDemo2 or maybe hystrix2 ... use proper name here.
you need "turbine.aggregator.clusterConfig=" which worked for me only when i used in CAPITAL i.e. HYSTRIX2
if you are using different management port on your service, "turbine.instanceUrlSuffix.HYSTRIX2=:/hystrix.stream
then this "turbine.instanceInsertPort=false" will disable default port insertion by turbine.. basically, you are telling eureka not to insert any port on its own when trying to search for hystrix.strem..
following are my properties.
#turbine.clusterNameExpression=new String('default')
#turbine.clusterNameExpression="'default'"
turbine.instanceInsertPort=false
turbine.appConfig=service1
turbine.aggregator.clusterConfig=SERVICE1
turbine.instanceUrlSuffix.SERVICE1=:51512/hystrix.stream
#turbine.ConfigPropertyBasedDiscovery.USER.instances=service1-host1.abc.com,service1-host2.abc.com
InstanceDiscovery.impl=com.netflix.turbine.discovery.EurekaInstanceDiscovery.class
#for high latencies
#turbine.InstanceMonitor.eventStream.skipLineLogic.enabled=false
and try the turbine stream on
http://host:port/turbine.stream?cluster=SERVICE1

RabbitMQ logins via LDAP failing

We are currently running RabbitMQ 3.5.6, and though it is able to successfully bind to the LDAP server, logins to the management UI via LDAP credentials are failing. I've been unable to track down the cause of this.
Our end goal is to have users be able to log into the RabbitMQ management UI with their LDAP credentials, and have RabbitMQ assign them permissions based on the groups that they are a member of in LDAP.
Upon login, both with a local account that I created for testing purposes and with my LDAP credentials I am presented with an internal server error:
Got response code 500 with body {"error":"Internal Server Error","reason":"{error,\n {try_clause,\n [{\"CN=rabbit,OU=System,OU=People,DC=domain,DC=tld\",\n \"LDAP_PASSWORD\"}]},\n [{rabbit_auth_backend_ldap,with_ldap,3,\n [{file,\"rabbitmq-auth-backend-ldap/src/rabbit_auth_backend_ldap.erl\"},\n {line,271}]},\n {rabbit_auth_backend_ldap,user_login_authentication,2,\n [{file,\"rabbitmq-auth-backend-ldap/src/rabbit_auth_backend_ldap.erl\"},\n {line,59}]},\n {rabbit_access_control,try_authenticate,3,\n [{file,\"src/rabbit_access_control.erl\"},{line,91}]},\n {rabbit_access_control,'-check_user_login/2-fun-0-',4,\n [{file,\"src/rabbit_access_control.erl\"},{line,77}]},\n {lists,foldl,3,[{file,\"lists.erl\"},{line,1262}]},\n {rabbit_mgmt_util,is_authorized,6,\n [{file,\"rabbitmq-management/src/rabbit_mgmt_util.erl\"},{line,121}]},\n {webmachine_resource,resource_call,3,\n [{file,\n \"webmachine-wrapper/webmachine-git/src/webmachine_resource.erl\"},\n {line,186}]},\n {webmachine_resource,do,3,\n [{file,\n \"webmachine-wrapper/webmachine-git/src/webmachine_resource.erl\"},\n {line,142}]}]}\n"}
The rabbitmq.config that I am currently using is below, followed by the log entries generated by RabbitMQ.
%% -*- mode: erlang -*-
[
{rabbit,
[{tcp_listeners, []},
{ssl_listeners, [{"10.7.232.1", 5672}]},
{log_levels, [{connection, info}, {channel, info}]},
{reverse_dns_lookups, true},
{ssl_options, [{certfile, "/usr/local/etc/rabbitmq/rmqs01.cer"},
{keyfile, "/usr/local/etc/rabbitmq/rmqs01.key"}]},
{auth_backends, [rabbit_auth_backend_ldap, rabbit_auth_backend_internal]},
{auth_mechanisms, ['PLAIN']}
]},
{rabbitmq_auth_backend_ldap,
[{servers, ["dc01.domain.tld", "dc02.domain.tld", "dc03.domain.tld"]},
%%{user_dn_pattern, "cn=${username},ou=People,dc=domain,dc=tld"},
{user_dn_pattern, []},
{use_starttls, true},
%% necessary for our ldap setup
{dn_lookup_attribute, "sAMAccountName"},
{dn_lookup_base, "OU=People,DC=domain,DC=tld"},
{dn_lookup_bind, [{"CN=rabbit,OU=System,OU=People,DC=domain,DC=tld", "rmqpassword"}]},
{port, 389},
{timeout, 30000},
{other_bind, [{"CN=rabbit,OU=System,OU=People,DC=domain,DC=tld", "rmqpassword"}]},
{log, network},
%% ACL testing
{resource_access_query,
{for, [{resource, exchange,
{for, [{permission, configure,
{ in_group, "OU=Systems,OU=People,DC=domain,DC=tld" } },
{permission, write, {constant, true}},
{permission, read, {constant, true}}
]}},
{resource, queue, {constant, true}} ]}}
]},
{rabbitmq_management,
%%{http_log_dir, "/var/log/rabbitmq/access.log"},
[{listener,
[{port, 15672},
{ip, "10.7.232.1"},
{ssl, true},
{ssl_opts,
[{certfile, "/usr/local/etc/rabbitmq/rmqs01.cer"},
{keyfile, "/usr/local/etc/rabbitmq/rmqs01.key"}
]}
]}
]}
].
=INFO REPORT==== 10-May-2016::10:17:47 ===
LDAP CHECK: login for username
=INFO REPORT==== 10-May-2016::10:17:47 ===
LDAP connecting to servers: ["dc01.domain.tld",
"dc02.domain.tld",
"dc03.domain.tld"]
=ERROR REPORT==== 10-May-2016::10:17:47 ===
webmachine error: path="/api/whoami"
{error,
{try_clause,
[{"CN=rabbit,OU=System,OU=People,DC=domain,DC=tld",
"rmqpassword"}]},
[{rabbit_auth_backend_ldap,with_ldap,3,
[{file,"rabbitmq-auth-backend-ldap/src/rabbit_auth_backend_ldap.erl"},
{line,271}]},
{rabbit_auth_backend_ldap,user_login_authentication,2,
[{file,"rabbitmq-auth-backend-ldap/src/rabbit_auth_backend_ldap.erl"},
{line,59}]},
{rabbit_access_control,try_authenticate,3,
[{file,"src/rabbit_access_control.erl"},{line,91}]},
{rabbit_access_control,'-check_user_login/2-fun-0-',4,
[{file,"src/rabbit_access_control.erl"},{line,77}]},
{lists,foldl,3,[{file,"lists.erl"},{line,1262}]},
{rabbit_mgmt_util,is_authorized,6,
[{file,"rabbitmq-management/src/rabbit_mgmt_util.erl"},{line,121}]},
{webmachine_resource,resource_call,3,
[{file,
"webmachine-wrapper/webmachine-git/src/webmachine_resource.erl"},
{line,186}]},
{webmachine_resource,do,3,
[{file,
"webmachine-wrapper/webmachine-git/src/webmachine_resource.erl"},
{line,142}]}]}

Resources