Multiple video call (n users) using Peerjs in React Native - reactjs

I have an application in which I am trying to get video chatting to work in React Native.
Used packages like react-native-webrtc and react-native-peerjs.
Created peer js server using Node Js.
One to One Video call is working fine with react native Peerjs. But, Now I want more than 2 users to be connected upto n users.
Is it possible to convert one to one video call to Multiple video call. Kindly let me know how Multiple video call can be achieved using Peer js and web rtc.
Here is my code for one to one video call:
Initialize webrtc and PeerJS:
const initialize = async () => {
const isFrontCamera = true;
const devices = await mediaDevices.enumerateDevices();
const facing = isFrontCamera ? 'front' : 'environment';
const videoSourceId = devices.find(
(device: any) => device.kind === 'videoinput' && device.facing === facing,
);
const facingMode = isFrontCamera ? 'user' : 'environment';
const constraints: MediaStreamConstraints = {
audio: true,
video: {
mandatory: {
minWidth: 1280,
minHeight: 720,
minFrameRate: 30,
},
facingMode,
optional: videoSourceId ? [{ sourceId: videoSourceId }] : [],
},
};
const newStream = await mediaDevices.getUserMedia(constraints);
setLocalStream(newStream as MediaStream);
console.log("************ Started ************");
// const io = socketio(SERVER_URL);
// io.connect();
console.log(SERVER_URL);
const io = socketio.connect(SERVER_URL, {
reconnection: true,
autoConnect: true,
reconnectionDelay: 500,
jsonp: false,
reconnectionAttempts: Infinity,
// transports: ['websocket']
});
io.on('connect', () => {
console.log("----------- Socket Connected -----------");
setSocket(io);
io.emit('register', username);
});
io.on('users-change', (users: User[]) => {
console.log("----------- New User - " + JSON.stringify(users) + " -----------");
setUsers(users);
});
io.on('accepted-call', (user: User) => {
setRemoteUser(user);
});
io.on('rejected-call', (user: User) => {
setRemoteUser(null);
setActiveCall(null);
Alert.alert('Your call request rejected by ' + user?.username);
navigate('Users');
});
io.on('not-available', (username: string) => {
setRemoteUser(null);
setActiveCall(null);
Alert.alert(username + ' is not available right now');
navigate('Users');
});
const peerServer = new Peer(undefined, {
host: PEER_SERVER_HOST,
path: PEER_SERVER_PATH,
secure: false,
port: PEER_SERVER_PORT,
config: {
iceServers: [
{
urls: [
'stun:stun1.l.google.com:19302',
'stun:stun2.l.google.com:19302',
],
},
],
},
});
peerServer.on('error', (err: Error) =>
console.log('Peer server error', err),
);
peerServer.on('open', (peerId: string) => {
setPeerServer(peerServer);
setPeerId(peerId);
io.emit('set-peer-id', peerId);
});
io.on('call', (user: User) => {
peerServer.on('call', (call: any) => {
//Alert.alert("PeerServer Call");
setRemoteUser(user);
Alert.alert(
'New Call',
'You have a new call from ' + user?.username,
[
{
text: 'Reject',
onPress: () => {
io.emit('reject-call', user?.username);
setRemoteUser(null);
setActiveCall(null);
},
style: 'cancel',
},
{
text: 'Accept',
onPress: () => {
io.emit('accept-call', user?.username);
call.answer(newStream);
setActiveCall(call);
navigate('Call');
},
},
],
{ cancelable: false },
);
call.on('stream', (stream: MediaStream) => {
setRemoteStream(stream);
});
call.on('close', () => {
closeCall();
});
call.on('error', () => { });
});
});
};
When a user call another user:
const call = (user: User) => {
if (!peerServer || !socket) {
Alert.alert('Peer server or socket connection not found');
return;
}
if (!user.peerId) {
Alert.alert('User not connected to peer server');
return;
}
socket.emit('call', user.username);
setRemoteUser(user);
try {
const call = peerServer.call(user.peerId, localStream);
call.on(
'stream',
(stream: MediaStream) => {
setActiveCall(call);
setRemoteStream(stream);
},
(err: Error) => {
console.error('Failed to get call stream', err);
},
);
} catch (error) {
console.log('Calling error', error);
}
};
Now, how should I call multiple user from the code below and how multiple streams have to be handled.
const call = peerServer.call(user.peerId, localStream);

Is it possible to convert one to one video call to Multiple video call
It's not possible to "convert" a one to one video call to "multiple" in a peer-to-peer architecture. In a p2p architecture with n participants, each participant will have a separate, one-to-one connection with the rest n-1 other participants.
I may possibly be misunderstanding your question, but if you're asking whether it's possible to establish n-1 connections for each participant, then the answer is yes. Here's how I would implement:
Anytime a new participant joins a session, extract their peer information. This is the peerId provided by the peer.js library.
Next, let the rest of the participants know about the presence of this new user. For this, you'll share this new participant's name, peerID and any other metadata with the rest of the participants in the room. This can be done by the signalling logic that you have implemented using socket.io.
Now going forward, you have 2 options:
The new participant could initiate the one-to-one peer connection with others in the room, OR,
The rest of the participants could initiate a one-on-one connection with the new participant.
Personally I prefer the first. So continuing the process:
Using the same signalling logic via socket.io, the rest of the participants will let the new user know about their presence by providing their own peer information and other metadata.
Once the new participant gets everyone's peer information, initiate a new peer connection using call.on('stream', callback) and start broadcasting their video.
On the recipient side, when a call is received along with the stream, you'll create a new video element in react-native, and bind the received media stream to this element. Which means, each participant will have n-1 video elements for streaming the media of n-1 other participants. The recipient also starts to broadcast their own video to the initiator of the call.
Here's a tutorial showing how this can be done using vanilla JavaScript, along with the github repository with source code.
Now, to answer the next question:
Kindly let me know how Multiple video call can be achieved using Peer js and webrtc.
This depends on the number of participants, where they lie geographically, browser/device limits, device computational power, and network bandwidth. So there are multiple factors involved which makes it tricky to give any specific number.
Browsers can place their own upper limits on the maximum number of connections possible, and there might be other values for Android and iOS. On chrome, the max theoretical limit is 500. If you're developing for Android, you may want to check here. But I couldn't manage to find much info on this.
Most practical applications involving WebRTC don't rely on a mesh architecture. Common implementations involve using an SFU, which takes multiple media streams and forwards them. A slightly more sophisticated technique is an MCU architecture, which combines all those media streams from multiple participants into a single one, and send that single stream to the rest of the participants.
I discuss this in some detail here:
https://egen.solutions/articles/how-to-build-your-own-clubhouse-part-2/#architectures-scaling-and-costs
Here's a nice article that explains the difference between SFU and MCU.

Related

UseEffect/useCallback is not triggering on very fast changes

I'm a Backend Dev and having limited knowledge in React still have to fix the problem
My project uses WebRTC for video calls. For signaling I'm using SignalR on my .NET backend.
On the frontend I have 2 classes:
signalRContext.tsx which holds an instance of HubConnection and listeners, onmessage is the relevant one.
const [currentSignal, setCurrentSignal] = useState<TCurrentSignal>
(InitialSignalR.currentSignal);
const initializeSignalListeners = (connection: HubConnection): void => {
console.log('START SIGNAL_R', connection);
connection.on('master', function (RoundInfo: IRoundInfo) {
console.log('MASTER', RoundInfo);
setCurrentSignal({ type: 'master', payload: RoundInfo });
});
connection.on('slave', function (RoundInfo: IRoundInfo) {
console.log('SLAVE', RoundInfo);
setCurrentSignal({ type: 'slave', payload: RoundInfo });
});
connection.on('message', function (message: TSignalRMessage) {
console.log('MESSAGE', message);
setCurrentSignal({ type: 'message', payload: message });
});
connection.on('endround', (payload) => {
console.log('END_ROUND');
setCurrentSignal({ type: 'endround', payload });
});
useRTCPeerConnection.ts which has the whole WebRtc relevant logic
import { useSignalRContext } from '../../../../core/contexts';
const {
signalrRRef,
currentSignal: { type, payload },
} = useSignalRContext();
useCallback(() => { //tried UseEffect as well
if (type === 'message') {
console.log('PAYLOAD', payload);
onMessage(payload as TSignalRMessage);
return;
}
}, [type, payload]);
My problem starts when WebRTC starts exchanging the ICE candidates and sends them sometimes twice per millisecond (see the last column).
The connection.on('message'... listener seems to be fast enough, I'm seeing all console.log('MESSAGE'... outputs in the console.
My problem is that the useCallback/useEffect logic is not firing on every payload change, like for 20 MESSAGE outputs I'm seeing 4-7 PAYLOAD outputs.
My assumption is that useEffect is simply not designed for such quick changes.
Is there any other concept better suitable to solve this problem or any improvement I could do here? Thinking on .NET I would just use the composition pattern and call the relevant method from peer connection class within the event handler in signalR class but not sure how to fix it here.
P.S. I've tried to wait until ICE candidates are gathered and sending them at once but the performance becomes not acceptable.

How to In App Purchases with ReactJS and Capacitor

I am using Capacitor to generate both the IOS and Android apps (not using Iconic) - this works well, but we are trying to implement IAP (for IOS only at this stage) and cannot figure it out.
I have followed various guides (https://ionicframework.com/docs/native/in-app-purchase-2 and https://purchase.cordova.fovea.cc/ and https://capacitorjs.com/docs/guides/in-app-purchases) but simply cannot get it working with React (not React Native)
Can someone point me in the right direction, or provide sample code?
You didn't describe what is going wrong, but here's a basic configuration that works for me on iOS.
I'm only including the part about the store:
index.tsx
import { IAPProduct, InAppPurchase2 } from '#ionic-native/in-app-purchase-2';
const startStoreEventListeners = () => {
if (isPlatformMobile()) {
document.addEventListener(
'deviceready',
() => {
const store = InAppPurchase2;
// Needed to use IAP + cordova plugins.
// Set debug messages.
// Default.
store.verbosity = store.QUIET;
// store.verbosity = store.DEBUG;
store.register([
{
id: subMonthly,
type: store.PAID_SUBSCRIPTION,
},
{
id: subAnnual,
type: store.PAID_SUBSCRIPTION,
},
]);
// Upon approval, verify the receipt.
store.when(subMonthly).approved((product: IAPProduct) => {
product.verify();
});
store.when(subAnnual).approved((product: IAPProduct) => {
product.verify();
});
// Upon receipt validation, mark the subscription as owned.
store.when(subMonthly).verified((product: IAPProduct) => {
product.finish();
});
store.when(subAnnual).verified((product: IAPProduct) => {
product.finish();
});
// Track all store errors
store.error((err: Error) => {
debugLog('Store Error', JSON.stringify(err));
});
// https://billing-dashboard.fovea.cc/setup/cordova
store.validator =
'https://validator.fovea.cc/v1/validate?appName=secret';
store.refresh();
startIonic();
},
false,
);
} else {
startIonic();
}
};
startStoreEventListeners();
serviceWorker.unregister();
Note that #ionic-native packages are deprecated and need to be converted.

SIP integration with call conference in JS

I am developing an Electron application with the integration of React.js as a front-end framework, which will be more like a calling application.
In that application-specific users can have multiple calls incoming, outgoing, mute | unmute calls, hold | unhold calls, etc.
For this functionality to be achieved we have our own sip server, and for integrating that SIP server, on the frontend we are using a library which is known as SIP.JS.
SIP.JS provides us mostly all the predefined functions to make a call, receive a call, mute, unmute, blind transfer, attended transfer, etc.
But when it comes to having a call conference, it doesn't have proper documentation for that.
SIP.JS specifies to us that we can use FreeSWITCH as well as ASTERISK in order to achieve the functionality, but with our specific requirements, no additional server needs to be integrated.
We have also referred to rfc documentation for the call conference, but no such progress was there.
So far what we did is:
Registered the userAgent
Code for Incoming call integrated
Code for outgoing calls integrated
multiple session handling is achieved, for multiple calls
mute | unmute, hold | unhold.
DTMF functionality
Blind Transfer, Attended Transfer
Ring all Devices
In this scenario of call conference, I guess we have to make changes in Incoming and outgoing session handling functions.
For registration and incoming call in context:
const getUAConfig = async (_extension, _name) => {
let alreadyLogin = '';
try {
alreadyLogin = 'yes';
if (alreadyLogin == 'yes') {
_displayname = _name;
_sipUsername = _extension;
_sipServer = 'SIP SERVER';
_sipPassword = 'SIP PASSWORD';
_wssServer = 'WSS SERVER;
const uri = UserAgent.makeURI('sip:' + _sipUsername + '#' + _sipServer);
const transportOptions = {
wsServers: 'WSS SERVER',
traceSip: true,
maxReconnectionAttempts: 1,
};
const userAgentOptions = {
uri: uri,
transportOptions: transportOptions,
userAgentString: 'App name',
authorizationPassword: _sipPassword,
sipExtension100rel: 'Supported',
sipExtensionReplaces: 'Supported',
register: true,
contactTransport: 'wss',
dtmfType: 'info',
displayName: _name,
sessionDescriptionHandlerFactoryOptions: {
peerConnectionOptions: {
rtcpMuxPolicy: 'negotiate',
iceCheckingTimeout: 1000,
iceTransportPolicy: 'all',
iceServers: [{ urls: 'stun:stun.l.google.com:19302' }],
},
},
};
userAgent = await new UserAgent(userAgentOptions);
const registerOptions = {
extraContactHeaderParams: [],
};
registerer = await new Registerer(userAgent, registerOptions);
registerer.stateChange.addListener((newState) => {
});
userAgent.start().then(async () => {
console.log('Connected with WebSocket.');
// Send REGISTER
await registerer
.register()
.then((request) => {
console.log('Successfully sent REGISTER, object is here');
dispatch({
type: USER_REGISTERED,
payload: true,
});
})
.catch((error) => {
console.log('Failed to send REGISTER');
});
});
return { userAgent, registerer };
} else {
return null;
}
} catch (error) {
console.log(error.message + '');
return null;
}
};
Outgoing functionality:
const dilaerFun = (inputNumber, userAgentInfo) => {
var session;
var uri = UserAgent.makeURI(
`URI which we wanna call (sip number)`
);
session = new Inviter(userAgentInfo, uri);
session
.invite()
.then((request) => {
console.log('Successfully sent INVITE');
sessionInfoAdd(session);
session.stateChange.addListener(async (state) => {
switch (state) {
case 'Established':
setMissedStatus(null);
console.log('established outgoing....');
//outgoing call log-----
const mediaElement = document.getElementById(
`mediaElement${session._id}`
);
const remoteStream = new MediaStream();
session.sessionDescriptionHandler.peerConnection
.getReceivers()
.forEach((receiver) => {
if (receiver.track) {
remoteStream.addTrack(receiver.track);
}
});
mediaElement.srcObject = remoteStream;
mediaElement.play();
break;
case 'Terminated':
console.log('terminated');
dispatch({
type: DEMO_STATE,
payload: session._id,
});
break;
default:
break;
}
});
})
.catch((error) => {
console.error(' Failed to INVITE');
console.error(error.toString());
});
};
Array of sessions are maintained by:
const sessionInfoAdd = (session) => {
dispatch({
type: SESSION_STORE,
payload: session,
});
};
Variable in which all sessions are stored is:
sessionInfo:[]
NOTE: getUAConfig() is called as soon as the application is started.
dialerFun() is called when we want to dial a specific number.
sessionInfoAdd() is called in both getUAConfig and dialerFun, as they are codes for incoming and outgoing calls.
when sessionInfoAdd() is triggered, the particular session which we get in return is added in the sessionInfo (Array) for the maintenance of sessions.
SIP.JS is just a library so you will have to get the conference setup on the FreeSWITCH or Asterisk (FreeSWITCH is the better in my opinion)
Doing this is fairly straight forward, at your app level you need a way to get calls across to the box after checking the details like access ID and any auth you want to add, (like a PIN.)
Once you have that done, you can forward that to an extension specifically set for conferencing or have a dynamic conference setup by send from the app towards a specific gateway/dialplan to do this.
The FreeSWITCH software has a steep learning curve on it but this helped me when I was doing something similar: https://freeswitch.org/confluence/display/FREESWITCH/mod_conference
You can also code you own conf if you wish.

Google Calendar API returning 403 for some users

I have integrated a small feature that reads the calendar events from each member of my organization's google calendar. It works for the majority but a couple get a 403 returned.
I think this is because they have more than one google account connected to the calendar maybe and I don't have permission to read those other accounts. This is the only reason I can think of. Hope someone can help out here - really stuck for a fix.
I am NOT using 'readonly' scopes. Below is the react function which loads the calendar events for the user.
const loadCalendar = () => {
window.gapi.load('client:auth2', () => {
console.log("fetching calendar for: ", userEmail);
gapi.client.init({
apiKey: "xyz",
clientId: "xyz",
discoveryDocs: ["https://www.googleapis.com/discovery/v1/apis/calendar/v3/rest"],
scope: "https://www.googleapis.com/auth/calendar",
})
gapi.client.load('calendar', 'v3', () => {
gapi.auth2.getAuthInstance().then(() => {
gapi.client.calendar.events.list({
'calendarId': 'primary',
'timeMin': (new Date()).toISOString(),
'showDeleted': false,
'singleEvents': true,
'maxResults': 12,
'orderBy': 'startTime',
}).then(({ result }) => {
let filteredArr = [];
result.items.forEach(item => {
let obj = {
key: item.id,
attendees: item.attendees,
start: item.start["date"] ? item.start["date"] : new Date(item.start["dateTime"]).toLocaleString(),
summary: item.summary,
zoomLink: item.location
}
filteredArr.push(obj);
})
setCalendarEvents(filteredArr)
})
})
})
});
}
After reviewing the script I understand that this script works for every user in your organization, except two particular users. In that situation I strongly recommend you to create a Workspace domain-wide delegation of authority. With that operation you will create an extra account that can access every user data in your organization, so you could access those two users calendars. The only required update is to use the new account credentials in your script. Please check the linked examples and write back if it doesn't work in your scenario.

PWA Push Notification not showing when app is not running

I'm developing a Progressive Web App with React that gets notifications when a new offer has been added to the DB. Everything works fine, I open the web, asks the user to grant permissions to enable notifications, we allow them, install the PWA, run it, add a new offer in the DB, and the a notification with the new offer gets displayed (Chrome + Windows 10).
But the issue is I don't get any notifications if the PWA is not running.. I would have thought the service worker is running in the background even if the PWA is closed. What am I missing?
here is my notifyNewOffer function in my notifications.ts file
function notifyNewOffer(newOffer: Offer) {
if ('serviceWorker' in navigator) {
const options = {
body: newOffer.subheading,
icon: './logo192.png',
image: './static/media/placeholder-offer.1bcbf040.png',
vibrate: [100, 50, 200],
badge: './favicon.ico',
tag: 'new-offers',
renotify: true,
actions: [
{ action: 'confirm', title: 'Check offer', icon: '' },
],
};
navigator.serviceWorker.ready.then(swreg => {
swreg.showNotification(newOffer.heading, options);
});
} else {
console.log('no serviceWorker');
}
}
And this is how I call it:
function addedOfferSubs<T>(setOffers: (offers:any) => void) {
// #ts-ignore
const subscription = API.graphql(graphqlOperation(addedOffer)).subscribe({
next: async (eventData: SubscriptionValue<T>) => {
const newOffer = (eventData.value.data as any).addedOffer;
await indexedDb.createObjectStore('offers', 'id'); // Opens db. Will create the table offers only if it doesnt already exist
await indexedDb.putValue('offers', newOffer); // Adds new offer
// Push notification
notifyNewOffer(newOffer);
// Set offers
const offersData = await getOffersFromIdb();
setOffers(offersData);
},
});
return () => subscription.unsubscribe()
}
Any ideas ?
Thanks very much
In order for notifications to appear when the app isn't open, you'll need to use Web Push as well. Web push allows you to send a notification from your server to device. When the push arrives on the device, it wakes up the service worker, and the notification is shown there.
Instructions for setting up Web Push & Notifications are available at https://developers.google.com/web/fundamentals/push-notifications

Resources